-
Notifications
You must be signed in to change notification settings - Fork 138
added initial telemetry work #386
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
95bd528
5430b78
d7bf70d
9cc6c8b
00b76b7
5f85940
5c831e8
b489291
5b3979f
9f5f317
c4267fc
99988fc
8c126a7
5e459f7
9549e0d
3336002
f5d9139
7cebc64
ab02383
b1628cd
6a902d7
353dba8
5a82b30
52a8318
766e10a
d2bf643
53132d1
ec5208f
ca9df67
9f0d61d
e59e34c
5797a30
cfafa97
0f09f7e
16e6a9b
070b188
b317564
88d8da3
1f1d8da
7f57fae
2e09a11
1d27b19
2d270df
5d7c1f9
91f8338
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,47 +14,53 @@ jobs: | |
matrix: | ||
python-version: [2.7, 3.4, 3.5, 3.6, 3.7, 3.8] | ||
framework: | ||
- FLASK_VERSION=0.10.1 Werkzeug\>=0.7,\<1.0 | ||
- FLASK_VERSION=0.11.1 Werkzeug\>=0.7,\<1.0 | ||
- FLASK_VERSION=0.12.4 Werkzeug\>=0.7,\<1.0 | ||
- FLASK_VERSION=1.0.2 | ||
- TWISTED_VERSION=15.5.0 treq==15.1.0 zope.interface==4.1.3 | ||
- TWISTED_VERSION=16.1.0 treq==16.12.0 zope.interface==4.1.3 | ||
- TWISTED_VERSION=16.2.0 treq==16.12.0 zope.interface==4.1.3 | ||
- TWISTED_VERSION=16.3.0 treq==16.12.0 zope.interface==4.2.0 | ||
- TWISTED_VERSION=16.4.0 treq==16.12.0 zope.interface==4.5.0 | ||
- TWISTED_VERSION=16.5.0 treq==16.12.0 zope.interface==4.5.0 | ||
- TWISTED_VERSION=16.6.0 treq==16.12.0 zope.interface==4.5.0 | ||
- TWISTED_VERSION=17.1.0 treq==16.12.0 zope.interface==4.5.0 | ||
- DJANGO_VERSION=1.11.20 | ||
- DJANGO_VERSION=2.0.13 | ||
- DJANGO_VERSION=2.1.7 | ||
- DJANGO_VERSION=2.1.15 | ||
- PYRAMID_VERSION=1.9.2 | ||
- PYRAMID_VERSION=1.10.4 | ||
- STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 | ||
- STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 | ||
- STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 | ||
- FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 | ||
- FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 | ||
- FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 | ||
- FLASK_VERSION=0.10.1 Werkzeug\>=0.7,\<1.0 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- FLASK_VERSION=0.11.1 Werkzeug\>=0.7,\<1.0 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- FLASK_VERSION=0.12.4 Werkzeug\>=0.7,\<1.0 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- FLASK_VERSION=1.0.2 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- TWISTED_VERSION=15.5.0 treq==15.1.0 zope.interface==4.1.3 mock==2.0.0 | ||
- TWISTED_VERSION=16.1.0 treq==16.12.0 zope.interface==4.1.3 mock==2.0.0 | ||
- TWISTED_VERSION=16.2.0 treq==16.12.0 zope.interface==4.1.3 mock==2.0.0 | ||
- TWISTED_VERSION=16.3.0 treq==16.12.0 zope.interface==4.2.0 mock==2.0.0 | ||
- TWISTED_VERSION=16.4.0 treq==16.12.0 zope.interface==4.5.0 mock==2.0.0 | ||
- TWISTED_VERSION=16.5.0 treq==16.12.0 zope.interface==4.5.0 mock==2.0.0 | ||
- TWISTED_VERSION=16.6.0 treq==16.12.0 zope.interface==4.5.0 mock==2.0.0 | ||
- TWISTED_VERSION=17.1.0 treq==16.12.0 zope.interface==4.5.0 mock==2.0.0 | ||
- DJANGO_VERSION=1.11.20 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- DJANGO_VERSION=2.0.13 mock==2.0.0 | ||
- DJANGO_VERSION=2.1.7 mock==2.0.0 | ||
- DJANGO_VERSION=2.1.15 mock==2.0.0 | ||
- PYRAMID_VERSION=1.9.2 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- PYRAMID_VERSION=1.10.4 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
exclude: | ||
- python-version: 2.7 | ||
framework: DJANGO_VERSION=2.0.13 | ||
framework: DJANGO_VERSION=2.0.13 mock==2.0.0 | ||
- python-version: 2.7 | ||
framework: DJANGO_VERSION=2.1.7 | ||
framework: DJANGO_VERSION=2.1.7 mock==2.0.0 | ||
- python-version: 2.7 | ||
framework: DJANGO_VERSION=2.1.15 | ||
framework: DJANGO_VERSION=2.1.15 mock==2.0.0 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=2.1.7 | ||
framework: DJANGO_VERSION=2.0.13 mock==2.0.0 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=2.1.15 | ||
framework: DJANGO_VERSION=2.1.7 mock==2.0.0 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=2.1.15 mock==2.0.0 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=2.1.15 mock==2.0.0 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=2.0.13 mock==2.0.0 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=2.1.15 | ||
framework: DJANGO_VERSION=2.1.7 mock==2.0.0 | ||
- python-version: 3.6 | ||
framework: DJANGO_VERSION=2.1.15 | ||
framework: DJANGO_VERSION=2.1.15 mock==2.0.0 | ||
- python-version: 3.7 | ||
framework: DJANGO_VERSION=2.1.15 | ||
framework: DJANGO_VERSION=2.1.15 mock==2.0.0 | ||
Comment on lines
-43
to
+63
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As a result of the comment above these changes are also not required. |
||
|
||
# twisted/treq setup.py allows: | ||
# Twisted < 18.7.0 on python < 3.7 | ||
|
@@ -95,83 +101,89 @@ jobs: | |
framework: TWISTED_VERSION=17.1.0 treq==20.4.1 zope.interface==4.3.0 | ||
|
||
- python-version: 2.7 | ||
framework: STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 2.7 | ||
framework: STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 2.7 | ||
framework: STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.12.12 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.12.13 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: STARLETTE_VERSION=0.14.2 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
|
||
- python-version: 2.7 | ||
framework: FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 2.7 | ||
framework: FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 2.7 | ||
framework: FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.40.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.50.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 | ||
framework: FASTAPI_VERSION=0.63.0 httpx==0.18.1 python-multipart==0.0.5 mock==3.0.5 | ||
Comment on lines
-98
to
+139
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as above. |
||
include: | ||
- python-version: 2.7 | ||
framework: FLASK_VERSION=0.9 | ||
- python-version: 3.3 | ||
framework: FLASK_VERSION=0.10.1 | ||
framework: FLASK_VERSION=0.10.1 mock==2.0.0 | ||
- python-version: 3.3 | ||
framework: FLASK_VERSION=0.11.1 | ||
framework: FLASK_VERSION=0.11.1 mock==2.0.0 | ||
- python-version: 3.3 | ||
framework: FLASK_VERSION=0.12.4 | ||
framework: FLASK_VERSION=0.12.4 mock==2.0.0 | ||
- python-version: 3.3 | ||
framework: FLASK_VERSION=1.0.2 | ||
framework: FLASK_VERSION=1.0.2 mock==2.0.0 | ||
- python-version: 3.3 | ||
framework: DJANGO_VERSION=1.6.11 | ||
framework: DJANGO_VERSION=1.6.11 mock==2.0.0 | ||
- python-version: 3.3 | ||
framework: DJANGO_VERSION=1.8.19 | ||
framework: DJANGO_VERSION=1.8.19 mock==2.0.0 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=1.7.11 | ||
framework: DJANGO_VERSION=1.7.11 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=1.8.19 | ||
framework: DJANGO_VERSION=1.8.19 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=1.9.13 | ||
framework: DJANGO_VERSION=1.9.13 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=1.10.8 | ||
framework: DJANGO_VERSION=1.10.8 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- python-version: 3.4 | ||
framework: DJANGO_VERSION=2.0.13 chardet==3.0.4 idna==2.7 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=1.8.19 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=1.9.13 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=1.8.19 | ||
framework: DJANGO_VERSION=1.10.8 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=1.9.13 | ||
framework: DJANGO_VERSION=2.0.13 mock==3.0.5 | ||
- python-version: 3.5 | ||
framework: DJANGO_VERSION=1.10.8 | ||
framework: DJANGO_VERSION=2.1.7 mock==3.0.5 | ||
- python-version: 3.7 | ||
framework: TWISTED_VERSION=18.9.0 treq==20.4.1 zope.interface==4.5.0 | ||
framework: TWISTED_VERSION=18.9.0 treq==20.4.1 zope.interface==4.5.0 mock==3.0.5 | ||
- python-version: 3.7 | ||
framework: TWISTED_VERSION=19.10.0 treq==20.4.1 zope.interface==4.6.0 | ||
framework: TWISTED_VERSION=19.10.0 treq==20.4.1 zope.interface==4.6.0 mock==3.0.5 | ||
- python-version: 3.7 | ||
framework: TWISTED_VERSION=20.3.0 treq==20.4.1 zope.interface==4.7.0 | ||
framework: TWISTED_VERSION=20.3.0 treq==20.4.1 zope.interface==4.7.0 mock==3.0.5 | ||
- python-version: 3.8 | ||
framework: TWISTED_VERSION=18.9.0 treq==20.4.1 zope.interface==4.5.0 | ||
framework: TWISTED_VERSION=18.9.0 treq==20.4.1 zope.interface==4.5.0 mock==3.0.5 | ||
- python-version: 3.8 | ||
framework: TWISTED_VERSION=19.10.0 treq==20.4.1 zope.interface==4.6.0 | ||
framework: TWISTED_VERSION=19.10.0 treq==20.4.1 zope.interface==4.6.0 mock==3.0.5 | ||
- python-version: 3.8 | ||
framework: TWISTED_VERSION=20.3.0 treq==20.4.1 zope.interface==4.7.0 | ||
framework: TWISTED_VERSION=20.3.0 treq==20.4.1 zope.interface==4.7.0 mock==3.0.5 | ||
Comment on lines
-138
to
+186
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Python 3.3+ includes the |
||
steps: | ||
- uses: actions/checkout@v2 | ||
with: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,6 @@ | |
import socket | ||
import sys | ||
import threading | ||
import time | ||
import traceback | ||
import types | ||
import uuid | ||
|
@@ -20,7 +19,8 @@ | |
import requests | ||
import six | ||
|
||
from rollbar.lib import events, filters, dict_merge, parse_qs, text, transport, urljoin, iteritems, defaultJSONEncode | ||
from collections import deque | ||
from rollbar.lib import events, filters, dict_merge, parse_qs, telemetry, text, transport, urljoin, iteritems, defaultJSONEncode, get_current_timestamp | ||
|
||
|
||
__version__ = '0.16.1' | ||
|
@@ -322,8 +322,12 @@ def _get_fastapi_request(): | |
'request_pool_connections': None, | ||
'request_pool_maxsize': None, | ||
'request_max_retries': None, | ||
'telemetry_queue_size': 50, | ||
} | ||
|
||
|
||
TELEMETRY_QUEUE = deque([], SETTINGS['telemetry_queue_size']) | ||
|
||
_CURRENT_LAMBDA_CONTEXT = None | ||
_LAST_RESPONSE_STATUS = None | ||
|
||
|
@@ -381,6 +385,18 @@ def init(access_token, environment='production', scrub_fields=None, url_fields=N | |
if SETTINGS.get('allow_logging_basic_config'): | ||
logging.basicConfig() | ||
|
||
queue_size = SETTINGS.get('telemetry_queue_size') | ||
if queue_size: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is buggy. You use the value as a number in other parts of the code but the value pulled from There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's fine. We have more settings that accept a number. Not sure why you would want to pass a string when it expects a number. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Python does not support strong typing, and you can't assume a source of the configuration data for the SDKs. It's common to read configuration settings from a serialized text file or even from a keyboard.
I am not against accepting numbers but the context of use. If you noticed any other settings that can lead to any bug, you can fix them by the way. |
||
TELEMETRY_QUEUE = deque([], queue_size) | ||
|
||
if SETTINGS.get('log_telemetry'): | ||
formatter = SETTINGS.get('log_telemetry_formatter') | ||
telemetry.enable_log_telemetry(formatter) | ||
if SETTINGS.get('network_telemetry'): | ||
enable_req_headers = SETTINGS.get('enable_req_headers') | ||
enable_resp_headers = SETTINGS.get('enable_resp_headers') | ||
telemetry.enable_network_telemetry(enable_req_headers, enable_resp_headers) | ||
|
||
if SETTINGS.get('handler') == 'agent': | ||
agent_log = _create_agent_log() | ||
|
||
|
@@ -777,6 +793,7 @@ def _report_exc_info(exc_info, request, extra_data, payload_data, level=None): | |
_add_request_data(data, request) | ||
_add_person_data(data, request) | ||
_add_lambda_context_data(data) | ||
_add_telemetry(data) | ||
data['server'] = _build_server_data() | ||
|
||
if payload_data: | ||
|
@@ -857,6 +874,7 @@ def _report_message(message, level, request, extra_data, payload_data): | |
_add_request_data(data, request) | ||
_add_person_data(data, request) | ||
_add_lambda_context_data(data) | ||
_add_telemetry(data) | ||
data['server'] = _build_server_data() | ||
|
||
if payload_data: | ||
|
@@ -887,7 +905,7 @@ def _check_config(): | |
|
||
def _build_base_data(request, level='error'): | ||
data = { | ||
'timestamp': int(time.time()), | ||
'timestamp': get_current_timestamp(), | ||
'environment': SETTINGS['environment'], | ||
'level': level, | ||
'language': 'python %s' % '.'.join(str(x) for x in sys.version_info[:3]), | ||
|
@@ -1119,6 +1137,12 @@ def _add_request_data(data, request): | |
data['request'] = request_data | ||
|
||
|
||
def _add_telemetry(data): | ||
telemetry_data = list(TELEMETRY_QUEUE) | ||
if telemetry_data: | ||
data['body']['telemetry'] = telemetry_data | ||
|
||
|
||
def _check_add_locals(frame, frame_num, total_frames): | ||
""" | ||
Returns True if we should record local variables for the given frame. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All Python versions from the matrix except 2.7 have the
mock
module included in the standard library. We don't need to install it manually.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, but it does not hurt to install it, and just for that I would need to pull 2.7 out of this matrix and have a different definition.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMHO requiring the installation of unnecessary packages hurts quality.
Our other test suites solve it in this way:
pyrollbar/rollbar/test/test_rollbar.py
Lines 14 to 17 in f67253a
You should follow the pattern. In my opinion, this PR does not require any changes in the CI builds. The builds failed for a good reason.
The installation of the
mock
package for Python <3.3 is already implemented:pyrollbar/setup.py
Lines 20 to 24 in f67253a