The DAS amplitude simulator simulates a real DAS interrogators communication into an Equinor DAS streaming platform. This version simulates the back-end system of the streaming platform using Docker Compose.
This project is not open to code contributions.
We have implemented a simple Distributed Acoustic Sensing (DAS) simulator that queues up a configurable number of amplitudes that matches the number of loci for an imaginary fiber optic sensing cable. The samples are grouped in the time axis so that they represent a full "fiber vector shot". The data is sent trying to match the actual rate that a real interrogator would, but for this implementation only a rudimentary control is made for the actual timing of the send operation.
You can make adjustment to most variables by updating the environment that the simulator starts in. Check the defaults.env and defaults-remote-control.env files found in the same directory as this file.
A successful DAS acquisition is made by initially contacting the Stream initiator (INITIATOR_URL variable).
{
"SchemaVersion": "2.0",
"AcquisitionId": "c7f312d1-6b56-4b6d-9b25-8db20dfc71ec",
"OpticalPathUUID": "00528e45-06d0-4110-bba4-e904afe02657",
"PulseWidth": 100.50,
"PulseWidthUnit": "ns",
"MaximumFrequency": 5000.0,
"DasInstrumentBoxUUID": "00528e45-06d0-4110-bba4-e904aaa02657",
"SpatialSamplingInterval": 1.1,
"GaugeLength": 10.209524,
"NumberOfLoci": 4086,
"PulseRate": 10000.0,
"Custom": {
"AnyCustomStuffHere": "XYZ",
"SuchAsFirmwareId": "1.7.2-Rev5",
"OrSoftwareVersion": "0.1-Beta",
"OrIdentificationAboutVendorInternalProfiles" : "Profile-name.file"
},
"MeasurementStartTime": "2020-11-01T19:02:03.0Z",
"VendorCode": "Simulator",
"StartLocusIndex": 0
}Note that we only accept positive int values for StartLocusIndex, and that even with a StartLocusIndex > 0 the numbering of the actual channel should always start at 0. Note that AcquisitionId must be unique for each request. If you rerun the curl example, generate a new UUID value first.
Here is a curl command that can be run for testing the back-end handshake (contained in the dependson-services folder):
curl -X POST http://localhost:8080/api/v2/acquisition/start \
-H 'Content-Type: application/json' -H 'X-Api-Key: 1aa111a11aa11a0a1a1aa1111a1a1a1a' \
-d '{"SchemaVersion": "2.0", "AcquisitionId": "c7f312d1-6b56-4b6d-9b25-8db20dfc71ec", "OpticalPathUUID": "00528e45-06d0-4110-bba4-e904afe02657", "PulseWidth": 100.50, "PulseWidthUnit": "ns", "MaximumFrequency": 5000.0, "DasInstrumentBoxUUID": "00528e45-06d0-4110-bba4-e904aaa02657", "SpatialSamplingInterval": 1.1, "GaugeLength": 10.209524, "NumberOfLoci": 4086, "PulseRate": 10000.0, "Custom": {"AnyCustomStuffHere": "XYZ","SuchAsFirmwareId": "1.7.2-Rev5","OrSoftwareVersion": "0.1-Beta"}, "MeasurementStartTime": "2023-08-18T09:45:42.512505674Z", "VendorCode": "Simulator", "StartLocusIndex": 0}'The initiator responds with a configuration object that provides information on how to proceed sending data to the Kafka platform.
{
"bootstrapServers": "broker:29092",
"topic": "608931017-amp",
"schemaRegistryUrl": "http://schemaregistry:8081",
"numberOfPartitions": 2,
"partitionAssignments": {
"0":"0",
"1":"1",
"2":"0",
"3":"1",
"4":"0",
"5":"1",
......
}
}In production with multiple brokers, this value is a comma-separated list of broker addresses. A notable value here is the 'partition assignments' map". It tells the kafka producer part of the code to send specific loci (key) to a specific partition. It is very important that this is being followed as strict ordering of loci-data can only be guaranteed in Kafka within the same partition.
Control of the parameters of the process in this example is done via environment values.
| Setting | Default value | Description |
|---|---|---|
VARIANT_PLUGIN |
SimulatorBoxUnit |
The implementation producing data for simulation. Possible values are SimulatorBoxUnit and StaticDataUnit. The bean need to be available on classpath as it is loaded at runtime |
VENDOR_CODE |
Simulator |
The vendor code identifying the vendor |
ACQUISITION_START_VERSION |
V2 |
The schema version used. Can be one of V1,V2 |
INITIATOR_URL |
The API endpoint base URL for the stream initiation service. |
|
INITIATOR_API_KEY |
No value |
An API key that needs to be set in the service behind the initiator url e.g. 1aa111a11aa11a0a1a1aa1111a1a1a1a |
KAFKA_SERVER_OVERRIDE |
No value |
If the Kafka server is on a network that has a specific naming that has no meaning on the calling end (e.g. inside a docker container where you would use localhost:9092) you might get a service name. Here you can override the name with an IP address. |
SCHEMA_REGISTRY_URL_OVERRIDE |
No value |
If the Schema registry server is on a network that has a specific naming that has no meaning on the calling end (e.g. inside a docker container where you would use http://localhost:8081) you might get a service name. Here you can override the name with an IP address. |
DAS_PRODUCER_HTTP_CONNECT_TIMEOUT |
5s |
HTTP connect timeout for initiator/schema registry checks. |
DAS_PRODUCER_HTTP_READ_TIMEOUT |
10s |
HTTP read timeout for initiator/schema registry checks. |
DAS_PRODUCER_HEALTHCHECK_TIMEOUT |
5m |
Max time to wait for initiator/schema registry to become healthy. |
DAS_PRODUCER_HEALTHCHECK_INTERVAL |
5s |
Delay between health check attempts. |
DAS_PRODUCER_SHUTDOWN_SLEEP |
1s |
Grace period during shutdown. |
DAS_PRODUCER_KAFKA_CONFIG_CLIENT_ID |
Simulator |
|
DAS_PRODUCER_KAFKA_CONFIG_ACKS |
1 |
|
DAS_PRODUCER_KAFKA_CONFIG_BATCH_SIZE |
500000 |
|
DAS_PRODUCER_KAFKA_CONFIG_BUFFER_MEMORY |
33554432 |
|
DAS_PRODUCER_KAFKA_CONFIG_COMPRESSION_TYPE |
none |
|
DAS_PRODUCER_KAFKA_CONFIG_LINGER_MS |
0 |
|
DAS_PRODUCER_KAFKA_CONFIG_MAX_REQ_SIZE |
1048576 |
|
DAS_PRODUCER_KAFKA_CONFIG_MAX_BLOCK_MS |
60000 |
|
DAS_PRODUCER_KAFKA_CONFIG_MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION |
5 |
|
DAS_PRODUCER_KAFKA_CONFIG_SEND_BUFFER_BYTES |
-1 |
|
DAS_PRODUCER_KAFKA_SENDER_CLOSE_TIMEOUT |
1s |
Timeout for Kafka producer close during shutdown. |
DAS_PRODUCER_KAFKA_RELAY_SHUTDOWN_TIMEOUT |
10s |
Timeout for relay executor shutdown. |
REMOTE_CONTROL_ENABLED |
false |
Enable remote-control mode. When true, the simulator waits for an APPLY request before producing data. |
REMOTE_CONTROL_API_KEY |
1234-abcde (defaults-remote-control.env) |
Required API key for the simulator’s remote-control endpoints. Callers must send X-Api-Key. |
REMOTE_CONTROL_PROFILES_DIR |
remote-control-profiles |
Directory containing JSON profiles named <das-simulator-profile>.json. |
TIME_PACING_ENABLED |
true |
Enable pacing of production to match the simulated package timestamps. |
TIME_LAG_WARN_MS |
500 |
Log a warning when the simulator lags behind wall-clock by more than this threshold. |
TIME_LAG_DROP_MS |
2000 |
Drop a fiber shot when lag exceeds this threshold to catch up (0 disables dropping). |
SIMULATOR_TIMING_LOG_INTERVAL |
30s |
Interval between timing summary logs for SimulatorBoxUnit. |
STATIC_TIMING_LOG_INTERVAL |
30s |
Interval between timing summary logs for StaticDataUnit. |
Any environment variable that starts with 'DAS_PRODUCER_KAFKA_CONFIG_' will be applied to the Kafka client configuration: http://kafka.apache.org/documentation.html#producerconfigs
When remote control is enabled, a valid X-Api-Key is required (defaults-remote-control.env provides a demo key).
Remote control uses JSON "profiles" stored on disk. A profile is a full DAS acquisition JSON object stored as
REMOTE_CONTROL_PROFILES_DIR/<profile-id>.json.
Profile lifecycle:
-
Creating a new configuration starts from the interrogator unit: it produces the acquisition JSON that defines the configuration. In this simulator, that configuration is represented by adding a new profile JSON file.
-
Switching back to a previously used configuration is done by sending an APPLY request that references the stored profile id.
APPLY behavior:
-
The APPLY request body must be JSON.
-
The simulator reads only
Custom.das-simulator-profilefrom the request body to select a profile file. All other fields in the APPLY request body are ignored. -
The simulator loads the matching profile from
REMOTE_CONTROL_PROFILES_DIRand uses that configuration to start producing data. -
A new
AcquisitionIdandMeasurementStartTimeare generated on every APPLY (including switches), so stored profile values are not reused.
Custom JSON (vendor-owned):
The Custom object inside a DAS acquisition payload is treated as vendor-owned, opaque JSON. The back-end system does
not interpret or modify the contents of Custom, except for reading Custom.das-simulator-profile in the incoming
APPLY request to resolve the profile file. This makes Custom the place to store stable identifiers
and metadata that can be used to map a switch request back to a previously used configuration.
The vendor can store any valid JSON under Custom (strings, numbers, booleans, objects, arrays).
Typical examples of vendor data stored under Custom include:
-
Vendor-specific acquisition identifiers (distinct from
AcquisitionId, which is regenerated per APPLY). -
Fiber optic switch information (for example, switch unit id and port number), if applicable.
-
Internal profile/configuration identifiers used by the interrogator unit.
Example Custom object in a profile file:
"Custom": {
"vendorAcquisitionId": "ACQ-2026-000123",
"fiberSwitch": { "unit": "SW-1", "port": 7 },
"interrogatorConfigId": "profile-A"
}When a profile is applied, the simulator forwards the profile JSON to the stream initiator, leaving Custom
unchanged.
STOP sends a best-effort stop signal to stream initiator for the last applied acquisition id. Example profiles
are included in the remote-control-profiles folder.
This step-by-step example starts the simulator with the random producer (SimulatorBoxUnit), then issues APPLY, switch(using APPLY), and STOP calls using curl.
1) Start the dependencies (Kafka, schema registry, streaminitiator). The dependson-services/.env
file configures the announced broker/schema registry addresses and the Kafka advertised host so
host-based clients get localhost endpoints:
docker-compose --env-file ./dependson-services/.env -f ./dependson-services/compose-kafka.yml up -dOptional: KafkaBat UI (port 5555) The stack also includes the KafkaBat UI so you can inspect topics, partitions, consumer groups, and message flow while the simulator is running. This is useful for quick validation of sequencing and volume without installing local Kafka tooling.
Open http://localhost:5555 after the compose stack is up.
2) Build and run the simulator in remote-control mode:
./mvnw clean package -DskipTests
source ./defaults-remote-control.env
export VARIANT_PLUGIN=SimulatorBoxUnit
java -jar ./das-producer/target/das-producer-<version>.jarIf you see java.net.UnknownHostException: broker, ensure dependson-services/.env contains:
ANNOUNCED_BROKERS_INGRESS=localhost:9092
ANNOUNCED_SCHEMAREGISTRY=http://localhost:8081
KAFKA_ADVERTISED_HOST=localhostand restart the docker-compose stack.
3) Start the simulator (APPLY) using the first profile:
curl -X POST http://localhost:9091/api/acquisition/apply \
-H 'Content-Type: application/json' \
-H 'X-Api-Key: 1234-abcde' \
-d '{"Custom":{"das-simulator-profile":"019c0d85-dc19-7dfe-859e-8d0c066f3c46"}}'4) Switch to a different profile (APPLY with a new profile id):
curl -X POST http://localhost:9091/api/acquisition/apply \
-H 'Content-Type: application/json' \
-H 'X-Api-Key: 1234-abcde' \
-d '{"Custom":{"das-simulator-profile":"019c0d85-2485-7ace-849c-e0e2be35c473"}}'5) Stop the simulator:
curl -X POST http://localhost:9091/api/acquisition/stop \
-H 'Content-Type: application/json' \
-H 'X-Api-Key: 1234-abcde' \
-d '{}'Note: You can also send the full profile JSON in the APPLY request body (for convenience), but the simulator
still loads the profile from disk and ignores all request fields other than Custom.das-simulator-profile
(including any extra vendor metadata under Custom). To change what gets forwarded to the stream initiator,
edit the profile JSON file in REMOTE_CONTROL_PROFILES_DIR.
Remote-control endpoints return structured error responses when requests are invalid or unauthorized.
| Error code | HTTP status | When it happens |
|---|---|---|
RC-400 |
400 |
Invalid request payload (missing/invalid JSON or required fields). |
RC-401 |
401 |
Missing or invalid X-Api-Key. |
RC-404 |
404 |
Requested acquisition id not found when stopping. |
RC-500 |
500 |
Unexpected internal error during remote-control handling. |
Details for ProdML can be found here: http://w3.energistics.org/energyML/data/prodml/v2.0/doc/prodml_schema_overview.html
Note:
You need Java 21 and you need docker and docker-compose installed to compile and run this.
Run the following commands in the same directory as this readme.adoc file (for Linux users). This starts a DAS producer that produces immediately (non-remote mode). For remote-control mode, see the "Remote-control walkthrough (docker-compose)" section above.
./mvnw clean install -DskipTests
docker-compose --env-file ./dependson-services/.env -f ./dependson-services/compose-kafka.yml up
source ./defaults.env
java -jar ./das-producer/target/das-producer-<version>.jarEmulation of an interrogator with random data.
Config name: SimulatorBoxUnit Config Options:
Setting |
Default value |
Description |
BOX_UUID |
No value |
the id (UUID) assigned by the stream initiator server. |
OPTICAL_PATH_UUID |
No value |
the id (UUID) assigned to the optical fiber where the data is requisitioned. |
GAUGE_LENGTH |
10.209524 |
as pr ProdML |
SPATIAL_SAMPLING_INTERVAL |
1.1 |
as pr ProdML - distance between data channel(loci) on the fiber. |
PULSE_WIDTH |
100.50 |
as pr ProdML - laser pulse width |
START_LOCUS_INDEX |
0 |
the point (index as in integer) where locus (channel) 0 is on the fiber. |
PULSE_RATE |
10000 |
as pr ProdML - laser pulse rate. |
MAX_NYQ_FREQ |
5000 |
as pr ProdML - max frequency available in signal (Nyquist). |
MIN_NYQ_FREQ |
0 |
as pr ProdML - min frequency available in signal (Nyquist). |
NUMBER_OF_LOCI |
No value |
The total number of data channels that will be delivered |
DISABLE_THROTTLING |
0 |
Ignore the time interval between data as pr. sampling frequency and package size. Instead deliver data as fast as possible. |
PACKAGE_SIZE |
8192 |
The number of amplitude delivered pr Kafka message. |
NUMBER_OF_SHOTS |
-1 |
Will override value set on SECONDS_TO_RUN to have a predefined number of shots sent |
SECONDS_TO_RUN |
240 |
Use this to limit the time the simulator should run. Can be overridden by NUMBER_OF_SHOTS |
START_TIME_EPOCH_SECOND |
0 |
Adjusts the first package timestamp. |
AMPLITUDE_DATA_TYPE |
float |
Values allowed, "float" or "long" |
TIME_PACING_ENABLED |
true |
Enable pacing of production to match the simulated package timestamps. |
TIME_LAG_WARN_MS |
500 |
Log a warning when the simulator lags behind wall-clock by more than this threshold. |
TIME_LAG_DROP_MS |
2000 |
Drop a fiber shot when lag exceeds this threshold to catch up (0 disables dropping). |
SIMULATOR_TIMING_LOG_INTERVAL |
30s |
Interval between timing summary logs. |
This variant emulates an interrogator that produces static data, which is useful for testing to ensure that known data is processed as expected. Config name: StaticDataUnit Config Options:
Setting |
Default value |
Description |
BOX_UUID |
No value |
the id (UUID) assigned by the stream initiator server. |
OPTICAL_PATH_UUID |
No value |
the id (UUID) assigned to the optical fiber where the data is requisitioned. |
NUMBER_OF_LOCI |
No value |
The total number of data channels that will be delivered |
PACKAGE_SIZE |
8192 |
The number of amplitude delivered pr Kafka message. |
DISABLE_THROTTLING |
0 |
Ignore the time interval between data as pr. sampling frequency and package size. Instead deliver data as fast as possible. |
NUMBER_OF_SHOTS |
-1 |
Will override value set on SECONDS_TO_RUN to have a predefined number of shots sent |
SECONDS_TO_RUN |
120 |
Use this to limit the time the simulator should run. Can be overridden by NUMBER_OF_SHOTS |
START_TIME_EPOCH_SECOND |
0 |
Adjusts the first package timestamp. |
AMPLITUDE_DATA_TYPE |
float |
Values allowed, "float" or "long" |
TIME_PACING_ENABLED |
true |
Enable pacing of production to match the simulated package timestamps. |
TIME_LAG_WARN_MS |
500 |
Log a warning when the simulator lags behind wall-clock by more than this threshold. |
TIME_LAG_DROP_MS |
2000 |
Drop a fiber shot when lag exceeds this threshold to catch up (0 disables dropping). |
STATIC_TIMING_LOG_INTERVAL |
30s |
Interval between timing summary logs. |
Fiber optic path UUIDs:
00528e45-06d0-4110-bba4-e904afe02657
9f79c244-1fec-4c78-83f9-e4b001f1c40f
966a5bdb-c170-4c5b-b84c-6cfc201b3654
4ab4497f-a227-4347-85cf-35a47a8d6fde
083a625f-5f86-4ff2-8f07-11156d305262
20e963bb-5790-4f30-a8cc-6ee6d8c43977
fbbc72c7-e915-4a9f-ab0b-31d5eb61e459
300b8ec2-b809-47bc-9b9c-844400a08f0a
11708574-3e9b-43cb-bd88-cd092cf55dd1
0e3212e7-42d2-4862-978a-5f424772cec1
1dc1c586-3136-45fa-aeba-7a04aaa8a6d3
a6f7bad1-4873-4cb7-8069-ef808365d454
f6d8b766-7246-4bd6-aafb-95c4cd0da91c
9e554c66-faae-43cb-a684-c7cc4dada609
8f52928e-fd26-438f-82a8-4cb35d139de6
622fba2c-9db3-4427-8330-69b56949fdd8
39f18236-7be1-44a2-aa43-18b1c80729dd
11760dd4-c484-4ec8-bb78-963416841412
f0cd6585-fcd8-412f-b44f-698c0c3a9ac3
4409eac5-ffe7-4433-9e5c-d0e0e1c2f1b8
d55a69cc-a3e8-4c95-be94-6dcf3b1c04b6
6d31c0c8-e6bd-4229-8920-84726d2f4cdf
81d75148-cfa1-48b5-9743-cff7e815a257
125109a7-406f-49ba-8839-c7002c735a55
e8b1f64b-1de5-47eb-97bc-7d645bc97c50
e8ce7b82-c4c5-4cfd-820a-06dc73ea3c72
b63f430f-e20a-4930-8613-3e07c5c4d51e
f293fe65-80ce-4846-8b3e-bab530ecc083
b2b9f48e-b8d7-4a1d-9ddf-89fd1c913d94
4ef35ba7-6167-4f01-be50-6df5fc7dc7e5
e56af441-40c6-4d82-a664-5c5c7cea2c6d
2f985cc0-8ada-4428-8554-912b72d58e8a
1d01ff83-c751-4a45-b1f9-6dc9056290f8
2585e913-9d3e-4b33-b187-f6074d4e7bad
99ec2d16-1d54-4766-8351-0398e903f1db
843fb17a-cdfa-47af-8812-aef791697600
ec9e0f9f-7100-4207-81df-b1eabf99bbda
28a795d5-6250-42ca-94b9-e1803299199f
dd6184a6-a25f-4bfd-94f6-abab8412812a
70c58966-dc28-4182-829e-532a5e6ae23b
326c9650-27fe-4c72-96af-6cef612a370a
e22c9119-027c-438f-b8fa-51f23cbb5cb0
61edef00-17c1-4777-854f-a54bbcc900fb
c166895f-6c64-47b7-9e35-e68bd0d41948
11763a71-627f-4c25-a955-a8b59301e536
8f0c8e8e-2ed6-4e4b-9e19-3309b6af184a
2ebcc749-3b37-4fb8-8234-4a56d918a88c
1118e763-9d7c-4466-b126-0d64aee850a2
a24c5a9e-510f-45ef-8112-3c6e6469e9c4Simulator box UUIDs:
00528e45-06d0-4110-bba4-e904aaa02657
6fc8a265-1ec5-4daf-b6b5-f7649af2a07a
1deb5d57-2fb9-418a-990b-4cf7252a0450
1deb5d57-2fb9-418a-990b-4cf7252a0451
1deb5d57-2fb9-418a-990b-4cf7252a0452
1deb5d57-3fb9-418a-990b-4cf7252a0451
1deb5d57-4fb9-418a-990b-4cf7252a0452