wgtool/scripting/README.md

# NPM Log Analysis Tools

High-performance security analysis tools for NPM (Nginx Proxy Manager) logs.

## Overview

This repository contains two versions of the NPM log analyzer:

1. **Bash Version** (`npm-log-analyzer.sh`) - Interactive menu-driven tool
2. **Go Version** (`npm-log-analyzer.go`) - High-performance command-line tool

## Features

### Security Pattern Detection
- **Critical Attacks**: SQL injection, XSS, shell/RCE, webshell uploads
- **High Priority**: Path traversal, WordPress hunting, backup harvesting
- **Reconnaissance**: Robots.txt requests, vulnerability scanners, error spam
- **Advanced**: SSRF, LFI/RFI, deserialization, template injection

### Analysis Capabilities
- Real-time log processing
- IP address analysis and geolocation
- Attack pattern counting and categorization
- Comprehensive reporting
- Performance optimization for large log files

## Quick Start

### Go Version (Recommended)
```bash
# Build and run
make go

# Or manually
go build -o npm-log-analyzer-go npm-log-analyzer.go
./npm-log-analyzer-go
```

### Bash Version
```bash
# Make executable and run
chmod +x npm-log-analyzer.sh
./npm-log-analyzer.sh
```

## Performance Comparison

| Feature | Go Version | Bash Version |
|---------|------------|--------------|
| **Speed** | ~85 seconds for 260MB logs | ~2-3 minutes |
| **Memory** | Efficient streaming | Higher memory usage |
| **Features** | Command-line focused | Interactive menu |
| **Dependencies** | Single binary | Requires bash, grep, etc. |
| **Gzip Support** | ✅ Native | ❌ Limited |

## Recent Analysis Results

From the latest Go analysis (260MB of logs):

### Critical Findings
- **SQL Injection Attempts**: 378
- **Shell/RCE Attempts**: 2,693
- **Error Spam (404/403)**: 4,855

### Top Attack Sources
- **169.150.203.13**: 45,278 requests (suspicious high volume)
- **135.181.143.221**: 9,229 requests
- **97.120.203.58**: 3,405 requests

### Analysis Statistics
- **Processed Files**: 98
- **Total Lines**: 105,568
- **Unique IPs**: 959
- **Duration**: ~85 seconds

## Configuration

### Log Directory
Both tools expect NPM logs at: `/opt/stacks/npm/data/logs`

### Output Directory
Reports are saved to: `./npmlogs`

### Attack Patterns
Patterns are defined in the source code and can be customized:

```go
// Go version
var AttackPatterns = map[string]string{
    "sql_injection": `union|select|insert|drop|delete`,
    "xss":          `<script|javascript:|onload=|onerror=`,
    // ... more patterns
}
```

```bash
# Bash version
declare -A ATTACK_PATTERNS=(
    ["sql_injection"]="union|select|insert|drop|delete"
    ["xss"]="<script|javascript:|onload=|onerror="
    # ... more patterns
)
```

## Usage Examples

### Go Version
```bash
# Quick analysis
./npm-log-analyzer-go

# Build for different platforms
GOOS=linux GOARCH=amd64 go build -o npm-analyzer-linux npm-log-analyzer.go
```

### Bash Version
```bash
# Interactive menu
./npm-log-analyzer.sh

# Quick analysis only
echo "1" | ./npm-log-analyzer.sh
```

## Makefile Targets

```bash
make build-go      # Build Go version
make run-go        # Build and run Go version
make go            # Alias for run-go
make run-bash      # Run Bash version
make bash          # Alias for run-bash
make benchmark     # Compare performance
make clean         # Clean build artifacts
make help          # Show all targets
```

## Requirements

### Go Version
- Go 1.16+ (for native gzip support)
- No external dependencies

### Bash Version
- Bash 4.0+
- grep, find, awk, curl
- Read access to `/opt/stacks/npm/data/logs`

## Security Considerations

1. **Log Access**: Both tools require read access to NPM logs
2. **Network Access**: IP geolocation requires internet access (optional)
3. **File Permissions**: Output directories need write permissions
4. **Large Files**: Processing 260MB+ logs requires sufficient memory

## Troubleshooting

### Common Issues

**"Log directory not accessible"**
```bash
# Check if NPM is running
ls -la /opt/stacks/npm/data/logs/

# Check permissions
sudo ls -la /opt/stacks/npm/data/logs/
```

**"No output generated"**
- Large log files take time to process
- Go version is much faster for large files
- Check available memory and disk space

**"Permission denied"**
```bash
# Make scripts executable
chmod +x npm-log-analyzer.sh
chmod +x npm-log-analyzer-go

# Check output directory permissions
mkdir -p ./npmlogs
chmod 755 ./npmlogs
```

## Development

### Adding New Patterns
1. Edit the pattern definitions in the source code
2. Test with sample log data
3. Update documentation

### Performance Optimization
- Go version uses buffered I/O and streaming
- Bash version uses grep with timeouts
- Both versions filter internal IPs automatically

## License

This project is open source. Feel free to modify and distribute.

## Contributing

1. Test both versions with your log data
2. Report any issues or performance problems
3. Suggest new attack patterns or features
4. Submit pull requests for improvements