Troubleshooting

Troubleshooting Guide

Common issues and solutions for rudder2snipe deployment and operation.

🚨 Common Issues

Authentication Problems

Issue: 401 Unauthorized from Rudder API

ERROR: Could not obtain a valid response from Rudder API
Response: 401 Unauthorized

Causes and Solutions:

Invalid API Token

# Test token manually
curl -H "X-API-TOKEN: YOUR-TOKEN" https://rudder.example.com/rudder/api/latest/nodes

Verify token is correct
Check token hasn't expired
Ensure token has read permissions

Wrong API Endpoint
```
# Check URL format
curl https://rudder.example.com/rudder/api/info
```
- Remove trailing slashes from URL
- Verify HTTPS/HTTP protocol
- Check port number if custom
Token Permissions
- Log into Rudder web interface
- Go to Administration → API Accounts
- Verify token has "read" permissions for nodes

Issue: 403 Forbidden from Snipe-IT API

ERROR: Snipe-IT responded with error code: 403

Solutions:

Check API Key Permissions

# Test API key
curl -H "Authorization: Bearer YOUR-TOKEN" https://snipe.example.com/api/v1/hardware

Ensure key has create/edit permissions
Check if key is active and not expired

User Permissions in Snipe-IT
- Log into Snipe-IT web interface
- Check user permissions for the API key owner
- Ensure user can create assets and models

Issue: SSL Certificate Verification Failed

ERROR: SSL verification failed

Solutions:

Temporary Bypass (Not Recommended for Production)
```
python3 rudder2snipe --do_not_verify_ssl
```

Proper SSL Certificate Setup

# Update CA certificates (Ubuntu/Debian)
sudo apt update && sudo apt install ca-certificates

# CentOS/RHEL
sudo yum update ca-certificates

Custom CA Certificate

export REQUESTS_CA_BUNDLE=/path/to/ca-certificates.crt
python3 rudder2snipe

Connection Issues

Issue: Connection Timeout

ERROR: Connection timeout when trying to reach API

Diagnostic Steps:

Network Connectivity

# Test basic connectivity
ping rudder.example.com
ping snipe.example.com

# Test port connectivity
telnet rudder.example.com 443
telnet snipe.example.com 443

DNS Resolution

# Check DNS resolution
nslookup rudder.example.com
nslookup snipe.example.com

Firewall Rules

# Check if HTTPS traffic is allowed
curl -I https://rudder.example.com
curl -I https://snipe.example.com

Issue: Connection Refused

ERROR: Connection refused

Solutions:

Verify Service Status
- Check if Rudder service is running
- Check if Snipe-IT web server is running
- Verify correct ports are open

URL Configuration

# Correct format (no trailing slash)
[rudder]
url = https://rudder.example.com

[snipe-it]
url = https://snipe.example.com

Configuration Issues

Issue: Invalid Configuration File

ERROR: No valid settings.conf was found

Solutions:

Check File Location

# rudder2snipe looks for config in these locations:
ls -la /opt/rudder2snipe/settings.conf
ls -la /etc/rudder2snipe/settings.conf
ls -la ./settings.conf

Verify File Format

# Check for syntax errors
python3 -c "import configparser; c=configparser.ConfigParser(); c.read('settings.conf'); print('Config OK')"

File Permissions

chmod 644 settings.conf
chown rudder:rudder settings.conf

Issue: Missing Required Configuration Sections

ERROR: Missing or invalid settings in settings.conf

Solution: Ensure all required sections exist:

[rudder]
url = https://rudder.example.com
api_token = your-token

[snipe-it]
url = https://snipe.example.com
apikey = your-key
manufacturer_id = 1
defaultStatus = 1
computer_model_category_id = 1

[api-mapping]
name = hostname

API Rate Limiting

Issue: 429 Too Many Requests

ERROR: We've been rate limited. Use option -r to respect the built in Snipe IT API rate limit

Solutions:

Enable Rate Limiting
```
python3 rudder2snipe --ratelimited
```
Reduce Concurrent Operations
- Run during off-peak hours
- Use smaller batch sizes
- Implement delays between operations

Check Snipe-IT Rate Limit Settings

# Check current rate limit
curl -I -H "Authorization: Bearer YOUR-TOKEN" https://snipe.example.com/api/v1/hardware
# Look for X-RateLimit-Limit header

Data Mapping Issues

Issue: Custom Fields Not Updating

WARNING: Unable to update custom field _snipeit_field_1

Diagnostic Steps:

Verify Custom Field DB Name

# Get custom fields list
curl -H "Authorization: Bearer YOUR-TOKEN" https://snipe.example.com/api/v1/fields

Check Field Permissions
- Ensure field is not read-only
- Verify field type supports the data being sent

Debug Field Mapping

python3 rudder2snipe --debug --dryrun | grep "field_1"

Issue: Invalid Model Creation

ERROR: Could not create model for machine type

Solutions:

Check Required IDs

# Verify manufacturer ID exists
curl -H "Authorization: Bearer YOUR-TOKEN" https://snipe.example.com/api/v1/manufacturers

# Verify category ID exists
curl -H "Authorization: Bearer YOUR-TOKEN" https://snipe.example.com/api/v1/categories

Check Model Creation Permissions
- Ensure API user can create models
- Verify all required fields are provided

Performance Issues

Issue: Slow Execution

INFO: Sync taking longer than expected

Optimization Steps:

Enable Rate Limiting for Large Deployments
```
python3 rudder2snipe --ratelimited
```
Check System Resources
```
# Monitor during execution
htop
iotop
```

Network Latency Check

# Test API response times
time curl -H "Authorization: Bearer TOKEN" https://snipe.example.com/api/v1/hardware?limit=1

Database Performance (Snipe-IT)
- Check MySQL/PostgreSQL performance
- Review slow query logs
- Consider database indexing

User Assignment Issues

Issue: Users Not Found in Snipe-IT

INFO: User john.doe not found

Solutions:

Check User Mapping Configuration

[user-mapping]
rudder_api_field = properties username

Verify User Exists in Snipe-IT

curl -H "Authorization: Bearer TOKEN" "https://snipe.example.com/api/v1/users?search=john.doe"

Enable User Search

# Allow broader user search
python3 rudder2snipe --users

# Disable search for exact matches only
python3 rudder2snipe --users --users_no_search

Issue: Asset Checkout Failed

ERROR: Asset checkout failed for asset 1234

Diagnostic Steps:

Check Asset Status
- Verify asset status allows checkout
- Ensure asset is not already checked out

User Permissions

# Check if user exists and is active
curl -H "Authorization: Bearer TOKEN" https://snipe.example.com/api/v1/users/123

Manual Checkout Test
- Try checking out asset manually in Snipe-IT web interface
- Verify checkout policies

🔍 Debugging Procedures

Enable Debug Logging

# Maximum verbosity
python3 rudder2snipe --debug --dryrun

API Response Testing

Test Rudder API

# Basic node list
curl -H "X-API-TOKEN: YOUR-TOKEN" \
     "https://rudder.example.com/rudder/api/latest/nodes" \
     | python3 -m json.tool

# Specific node details
curl -H "X-API-TOKEN: YOUR-TOKEN" \
     "https://rudder.example.com/rudder/api/latest/nodes/node-id" \
     | python3 -m json.tool

Test Snipe-IT API

# Basic hardware list
curl -H "Authorization: Bearer YOUR-TOKEN" \
     "https://snipe.example.com/api/v1/hardware" \
     | python3 -m json.tool

# Search by serial
curl -H "Authorization: Bearer YOUR-TOKEN" \
     "https://snipe.example.com/api/v1/hardware/byserial/SERIAL123" \
     | python3 -m json.tool

# Get models
curl -H "Authorization: Bearer YOUR-TOKEN" \
     "https://snipe.example.com/api/v1/models" \
     | python3 -m json.tool

Log Analysis

Common Log Patterns

Successful Operation:

INFO:root:Processing entry 1 out of 50 - NodeID: node1 - NAME: server01
INFO:root:Successfully created asset ID 1234 for server01

Authentication Error:

ERROR:root:Could not obtain a valid response from API
ERROR:root:Response: 401 Unauthorized

Rate Limiting:

WARNING:root:Despite respecting the rate limit, we've still been limited
INFO:root:Sleeping for 2 seconds before retry

Field Mapping Issue:

WARNING:root:Unable to update field _snipeit_custom_1 with value "test"
DEBUG:root:Field mapping failed: KeyError 'custom_field'

Log Analysis Script

#!/bin/bash
# analyze-logs.sh

LOG_FILE="/var/log/rudder2snipe.log"

echo "=== rudder2snipe Log Analysis ==="
echo "Log file: $LOG_FILE"
echo

# Count operations
echo "Operations Summary:"
echo "  Created assets: $(grep -c "Successfully created asset" "$LOG_FILE")"
echo "  Updated assets: $(grep -c "Successfully updated" "$LOG_FILE")"
echo "  Errors: $(grep -c "ERROR" "$LOG_FILE")"
echo "  Warnings: $(grep -c "WARNING" "$LOG_FILE")"
echo

# Recent errors
echo "Recent Errors:"
grep "ERROR" "$LOG_FILE" | tail -5
echo

# Recent warnings
echo "Recent Warnings:"
grep "WARNING" "$LOG_FILE" | tail -5
echo

# Performance info
echo "Performance:"
START_TIME=$(grep "Starting" "$LOG_FILE" | tail -1 | cut -d' ' -f1-2)
END_TIME=$(grep "completed\|finished" "$LOG_FILE" | tail -1 | cut -d' ' -f1-2)
echo "  Last run: $START_TIME to $END_TIME"

# API call counts
API_CALLS=$(grep -c "API call" "$LOG_FILE")
echo "  API calls made: $API_CALLS"

Network Debugging

Capture API Traffic

# Monitor HTTP/HTTPS traffic
sudo tcpdump -i any -s 0 -w rudder2snipe.pcap host rudder.example.com or host snipe.example.com

# Analyze with tshark
tshark -r rudder2snipe.pcap -Y "http or ssl" -T fields -e frame.number -e ip.src -e ip.dst -e http.response.code

Test with Curl

#!/bin/bash
# api-test.sh

RUDDER_URL="https://rudder.example.com"
RUDDER_TOKEN="your-token"
SNIPE_URL="https://snipe.example.com"
SNIPE_TOKEN="your-token"

echo "Testing Rudder API..."
curl -w "Response time: %{time_total}s\nHTTP code: %{http_code}\n" \
     -H "X-API-TOKEN: $RUDDER_TOKEN" \
     "$RUDDER_URL/rudder/api/latest/nodes?limit=1"

echo -e "\nTesting Snipe-IT API..."
curl -w "Response time: %{time_total}s\nHTTP code: %{http_code}\n" \
     -H "Authorization: Bearer $SNIPE_TOKEN" \
     "$SNIPE_URL/api/v1/hardware?limit=1"

🔧 Performance Optimization

System Optimization

Memory Usage

# Monitor memory usage during sync
ps aux | grep rudder2snipe

# Check available memory
free -h

Disk I/O

# Monitor disk I/O
iotop -p $(pgrep -f rudder2snipe)

API Optimization

Batch Operations

Process assets in smaller batches
Implement pagination for large datasets
Use API filters to reduce data transfer

Caching

# Example: Cache model lookups
models_cache = {}

def get_model_id(model_name):
    if model_name not in models_cache:
        # API call to get model
        models_cache[model_name] = api_result
    return models_cache[model_name]

Database Optimization (Snipe-IT)

Index Optimization

-- Add indexes for common searches
CREATE INDEX idx_assets_serial ON assets(serial);
CREATE INDEX idx_assets_asset_tag ON assets(asset_tag);
CREATE INDEX idx_assets_updated_at ON assets(updated_at);

Configuration Tuning

# Snipe-IT .env optimizations
DB_CONNECTION_TIMEOUT=60
QUEUE_DRIVER=database
CACHE_DRIVER=redis

📊 Monitoring and Alerting

Health Check Script

#!/bin/bash
# health-check.sh

LOG_FILE="/var/log/rudder2snipe.log"
ALERT_EMAIL="[email protected]"

# Check if log file exists and is recent
if [ ! -f "$LOG_FILE" ]; then
    echo "ERROR: Log file not found" | mail -s "rudder2snipe Alert" "$ALERT_EMAIL"
    exit 1
fi

# Check log age (should be less than 25 hours for daily sync)
LOG_AGE_HOURS=$((($(date +%s) - $(stat -f %m "$LOG_FILE" 2>/dev/null || stat -c %Y "$LOG_FILE" 2>/dev/null)) / 3600))

if [ $LOG_AGE_HOURS -gt 25 ]; then
    echo "WARNING: Log file is $LOG_AGE_HOURS hours old" | mail -s "rudder2snipe Alert" "$ALERT_EMAIL"
fi

# Check for errors in recent logs
ERROR_COUNT=$(grep "ERROR" "$LOG_FILE" | wc -l)
if [ $ERROR_COUNT -gt 0 ]; then
    echo "ERROR: $ERROR_COUNT errors found in log" | mail -s "rudder2snipe Alert" "$ALERT_EMAIL"
fi

echo "Health check passed"

Nagios/Icinga Integration

#!/bin/bash
# check_rudder2snipe

LOGFILE="/var/log/rudder2snipe.log"
CRITICAL_HOURS=25
WARNING_ERRORS=5

if [ ! -f "$LOGFILE" ]; then
    echo "CRITICAL: Log file not found"
    exit 2
fi

# Check age
HOURS_SINCE=$(($(date +%s) - $(stat -c %Y "$LOGFILE" 2>/dev/null || stat -f %m "$LOGFILE" 2>/dev/null)) / 3600)

if [ $HOURS_SINCE -gt $CRITICAL_HOURS ]; then
    echo "CRITICAL: Last run $HOURS_SINCE hours ago"
    exit 2
fi

# Check errors
ERROR_COUNT=$(grep -c "ERROR" "$LOGFILE")

if [ $ERROR_COUNT -gt $WARNING_ERRORS ]; then
    echo "WARNING: $ERROR_COUNT errors in log"
    exit 1
fi

echo "OK: rudder2snipe running normally"
exit 0

🆘 Getting Help

Before Requesting Support

Check this troubleshooting guide
Search existing issues: GitHub Issues
Review FAQ: FAQ Page
Enable debug logging: --debug --dryrun

Creating Support Requests

Include the following information:

Environment Details

Operating system and version
Python version
rudder2snipe version
Rudder version
Snipe-IT version

Configuration (Sanitized)

# Remove actual tokens/keys
[rudder]
url = https://rudder.example.com
api_token = [REDACTED]

[snipe-it]
url = https://snipe.example.com
apikey = [REDACTED]
# ... other settings

Error Messages

Full error output
Relevant log entries
Debug output if available

Steps to Reproduce

Detailed steps that led to the issue
Expected vs actual behavior
Any workarounds attempted

Community Resources

GitHub Issues: Report bugs
GitHub Discussions: Community forum
Documentation: Complete guide

Next: FAQ

Troubleshooting

Troubleshooting Guide

🚨 Common Issues

Authentication Problems

Issue: 401 Unauthorized from Rudder API

Issue: 403 Forbidden from Snipe-IT API

Issue: SSL Certificate Verification Failed

Connection Issues

Issue: Connection Timeout

Issue: Connection Refused

Configuration Issues

Issue: Invalid Configuration File

Issue: Missing Required Configuration Sections

API Rate Limiting

Issue: 429 Too Many Requests

Data Mapping Issues

Issue: Custom Fields Not Updating

Issue: Invalid Model Creation

Performance Issues

Issue: Slow Execution

User Assignment Issues

Issue: Users Not Found in Snipe-IT

Issue: Asset Checkout Failed

🔍 Debugging Procedures

Enable Debug Logging

API Response Testing

Test Rudder API

Test Snipe-IT API

Log Analysis

Common Log Patterns

Log Analysis Script

Network Debugging

Capture API Traffic

Test with Curl

🔧 Performance Optimization

System Optimization

Memory Usage

Disk I/O

API Optimization

Batch Operations

Caching

Database Optimization (Snipe-IT)

Index Optimization

Configuration Tuning

📊 Monitoring and Alerting

Health Check Script

Nagios/Icinga Integration

🆘 Getting Help

Before Requesting Support

Creating Support Requests

Environment Details

Configuration (Sanitized)

Error Messages

Steps to Reproduce

Community Resources

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally