tee-worker

Tee-worker is the Masa component to scrape data from a secure TEE enclave. It uses the ego framework to build, run and sign the binary.

Want to help in development? check the DEVELOPMENT.md file.

Requirements

• Docker

Run

To run the tee-worker, use docker with our images. Our images have signed binaries which are allowed to be part of the network:

mkdir .masa
wget https://raw.githubusercontent.com/masa-finance/tee-worker/refs/heads/main/.env.example -O .masa/.env
# Edit .masa/.env with your settings

# Run the worker
docker run --device /dev/sgx_enclave --device /dev/sgx_provision --net host --rm -v $(PWD)/.masa:/home/masa -ti masaengineering/tee-worker:main

Container images

All tagged images are available here: https://hub.docker.com/r/masaengineering/tee-worker/tags

• Images with latest tag are the latest releases
• Every branch has a corresponding image with the branch name (e.g. main)

Docker compose

There is an example docker compose file to run the container with the appropriate environment variables.

docker-compose up

Testing Mode

For testing outside a TEE environment:

// Enable standalone mode
tee.SealStandaloneMode = true

// Create a new key ring and add a key for standalone mode (32 bytes for AES-256)
keyRing := tee.NewKeyRing()
keyRing.Add("0123456789abcdef0123456789abcdef")

// Set as the current key ring
tee.CurrentKeyRing = keyRing

Important Notes

1. All encryption keys must be exactly 32 bytes long for AES-256 encryption
   • The system validates that keys are exactly 32 bytes (256 bits) when added through the SetKey function
   • An error will be returned if the key length is invalid
   • Example valid key: "0123456789abcdef0123456789abcdef" (32 bytes)
2. The sealing mechanism uses the TEE's product key in production mode
3. Key rings help manage multiple encryption keys and support key rotation
4. Salt-based key derivation adds an extra layer of security by deriving unique keys for different contexts

API

The tee-worker exposes a simple http API to submit jobs, retrieve results, and decrypt the results.

SIG=$(curl localhost:8080/job/generate -H "Content-Type: application/json" -d '{ "type": "web-scraper", "arguments": { "url": "google" } }')

### Submitting jobs
uuid=$(curl localhost:8080/job/add -H "Content-Type: application/json" -d '{ "encrypted_job": "'$SIG'" }' | jq -r .uid)

### Jobs results
result=$(curl localhost:8080/job/status/$uuid)

### Decrypt job results
curl localhost:8080/job/result -H "Content-Type: application/json" -d '{ "encrypted_result": "'$result'", "encrypted_request": "'$SIG'" }'

Golang client

It is available a simple golang client to interact with the API:

import(
    . "github.com/masa-finance/tee-worker/pkg/client"
    "github.com/masa-finance/tee-worker/api/types"
)

func main() {

    clientInstance := NewClient(server.URL)

    // Step 1: Create the job request
    job := types.Job{
        Type: "web-scraper",
        Arguments: map[string]interface{}{
            "url": "https://google.com",
            "depth": 1,
        },
    }

    // Step 2: Get a Job signature. Send the signature somewhere to be executed, or execute it locally (see below)
    jobSignature, err := clientInstance.CreateJobSignature(job) 

	// Step 3: Submit the job signature for execution ( can be done locally or remotely )
	jobResult, err := clientInstance.SubmitJob(jobSignature)

    // Step 4a: Get the job result (decrypted)
    result, err := jobResult.GetDecrypted(jobSignature)

    // Alternatively, you can get the encrypted result and decrypt it later
    // Note: this can be forwarded to another party to decrypt the result

	// Step 4b.1: Get the job result (encrypted)
	encryptedResult, err := jobResult.Get()

    // Step 4b.2: Decrypt the result
    decryptedResult, err := clientInstance.Decrypt(jobSignature, encryptedResult)
}

Job types

The tee-worker currently supports 3 job types:

TODO: Add descriptions of the return values.

web-scraper

Scrapes a URL down to some depth.

Arguments

• url (string): The URL to scrape.
• depth (int): How deep to go (if unset or less than 0, will be set to 1).

twitter-scraper

Performs different types of Twitter searches.

Arguments

• type (string): Type of query (see below).
• query (string): The query to execute. Its meaning depends on the type of query (see below)
• count (int): How many results to return.
• next_cursor (int): Cursor returned from the previous query, for pagination (for those job types that support it).

Job types

Some job types now support cursor-based pagination. For these jobs:

• The get variants ignore the next_cursor parameter and retrieve the first count records quickly
• To paginate, first use an empty next_cursor to get initial results, then use the returned next_cursor in subsequent calls.

Jobs that return tweets or lists of tweets

• searchbyquery - Executes a query and returns the tweets that match. The query parameter is a query using the Twitter API query syntax
• getbyid - Returns a tweet given its ID. The query parameter is the tweet ID.
• getreplies - Returns a list of all the replies to a given tweet. The query parameter is the tweet ID.
• gettweets - Returns all the tweets for a given profile. The query parameter is the profile to search.
• gethometweets - Returns all the tweets from a profile's home timeline. The query parameter is the profile to search.
• getforyoutweets - Returns all the tweets from a profile's "For You" timeline. The query parameter is the profile to search.
• getbookmarks - Returns all of a profile's bookmarked tweets. The query parameter is the profile to search.

Jobs that return profiles or lists of profiles

• getprofilebyid / searchbyprofile - Returns a given user profile. The query parameter is the profile to search for.
• getfollowers / searchfollowers - Returns a list of profiles of the followers of a given profile. The query parameter is the profile to search.
• getfollowing - Returns all of the profiles a profile is following. The query parameter is the profile to search.
• getretweeters - Returns a list of profiles that have retweeted a given tweet. The query parameter is the tweet ID.

Jobs that return other types of data

• getmedia - Returns info about all the photos and videos for a given user. The query parameter is the profile to search.
• gettrends- Returns a list of all the trending topics. The query parameter is ignored.
• getspace- Returns info regarding a Twitter Space given its ID. The query parameter is the space ID.

telemetry

This job type has no parameters, and returns the current state of the worker. It returns an object with the following fields. All timestamps are given in local time, in seconds since the Unix epoch (1/1/1970 00:00:00 UTC). The counts represent the interval between the boot_time and the current_time. All the fields in the stats object are optional (if they are missing it means that its value is 0).

Note that the stats are reset whenever the node is rebooted (therefore we need the boot_time to properly account for the stats)

These are the fields in the response:

• boot_time - Timestamp when the process started up.
• last_operation_time - Timestamp when the last operation happened.
• current_time - Current timestamp of the host.
• stats.twitter_scrapes - Total number of Twitter scrapes.
• stats.twitter_returned_tweets - Number of tweets returned to clients (this does not consider other types of data such as profiles or trending topics).
• stats.twitter_returned_profiles - Number of profiles returned to clients.
• stats.twitter_returned_other - Number of other records returned to clients (e.g. media, spaces or trending topics).
• stats.twitter_errors - Number of errors while scraping tweets (excluding authentication and rate-limiting).
• stats.twitter_ratelimit_errors - Number of Twitter rate-limiting errors.
• stats.twitter_auth_errors - Number of Twitter authentication errors.
• stats.web_success - Number of successful web scrapes.
• stats.web_errors - Number of web scrapes that resulted in an error.
• stats.web_invalid - Number of invalid web scrape requests (at the moment, blacklisted domains).

Profiling

The tee-worker supports profiling via pprof. The TEE does not allow for profiling, so it can only be enabled when running in standalone mode.

There are two ways to enable profiling:

• Set ENABLE_PPROF to true.
• Send a POST request to /debug/pprof/enable (no body necessary)

There is currently no way to completely disable profiling short of restarting the tee-worker. However, you can send a POST request to /debug/pprof/disable which will disable the most resource-intensive probes (goroutine blocking, mutexes and CPU).

When profiling is enabled you will have access to the following endpoints, which you can use with the go tool pprof command:

/debug/pprof - Index page /debug/pprof/heap - Heap profile /debug/pprof/goroutine - Goroutine profile /debug/pprof/profile?seconds=XX - CPU profile during XX seconds /debug/pprof/block - Goroutine blocking /debug/pprof/mutex - Holders of contended mutexes

There are others, see the /debug/pprof index page for a complete list.

The /debug/pprof/trace?seconds=XX will give you an XX-second execution trace, which you can use via the go tool trace command.

For more information, see the official docs. This link also contains useful information.