Statement
A Note on Recent Security Breach Attempts and Fine-Tuning Claims Against NovaSpark
Recently, there's been an accusation made by Harald Schaefer at Comma circulating that IQ's NovaSpark driving model is a “short fine-tuned” version of comma.ai's Rebellious Hope model.
Let's get something straight: NovaSpark was trained independently and it is not a “fine-tune” or distillation of Rebellious Hope.
When we actually compare NovaSpark with Rebellious Hope, the findings contradict Comma’s claim of a short fine-tune with an unchanged policy, as shown below.
These findings result from comparing the released IQ_R3_step2000 NovaSpark checkpoint with Rebellious Hope, the model Comma claims NovaSpark was fine-tuned from.
0/355 of NovaSpark’s parameter tensors match Rebellious Hope (0/134 backbone tensors and 0/221 policy tensors). All 221 corresponding policy tensors differ as complete tensors, including after FP16 rounding.
NovaSpark’s vision backbone has approximately 2.2 times as many parameters. Rebellious Hope has 12 vision blocks, arranged 2/2/6/2 across four stages, while NovaSpark has 18, arranged 2/2/10/4, with feed-forward (MLP) expansion increased from 3× to 4×. NovaSpark has 15,364,288 vision parameters, compared with Rebellious Hope’s 7,017,792.
NovaSpark has 8,346,496 more learned parameters overall (56,032,080 versus 47,685,584), an increase of 17.5%, and its estimated computational workload is approximately 57% higher.
Overall weight correlation is essentially zero. Across approximately 47.6 million weights, the reported correlation is 0.0003. A separate comparison of 116 corresponding weight matrices, covering 41.8 million coefficients, shows a median Pearson correlation of around 0.000010 (1 indicates perfect positive correlation, 0 indicates no linear correlation.)
Relative to RH’s weight magnitudes, the measured weight differences are 104.5% for vision, 103.6% for point policy, 128.0% for off-policy and 126.8% for on-policy.
Every policy section independently shows near-zero weight correlation. The median correlations are −0.002027 for point policy, −0.000599 for off-policy and +0.000453 for on-policy.
Individual vision components are also nearly entirely uncorrelated. The stem, all four vision stages, downsampling layers, final convolution, and output projection have correlations with Rebellious Hope ranging from approximately −0.007 to +0.03.
The internal parameter distributions are dramatically different, as NovaSpark's normalization weights cluster around 0.99 vs RH's median of 0.26. Token-mixer center coefficients are approximately 0.01 in NovaSpark versus 1.0 in RH.
NovaSpark was exported using PyTorch’s older exporter with ONNX opset 18 and full-precision FP32 weights. RH was exported using Comma’s newer exporter with opset 20 and half-precision FP16 weights.
Comparing the vision stages in Netron, NovaSpark has four residual blocks in its final vision stage, compared to Rebellious Hope’s two. NovaSpark’s longer exported chain also reflects GELU activations expanded into individual arithmetic operations.
NovaSpark’s vision-to-policy junction also demonstrates significant differences in the preparation steps and shape operations.
NovaSpark’s temporal transformer blocks also show significant architectural differences.
NovaSpark also has a distinct output assembly.
NovaSpark also incorporates navigation in its training and uses a new input,
PolyRouteLine64(when available), withRouteLine16as a backup input only used when navigation is active. This helps NovaSpark remember navigation instructions over a much longer context window.
Additionally, the RunPod statement below shows our cloud GPU spending for NovaSpark’s data labeling with the teacher, pre training, post training, scoring, and testing.
Unfortunately, before we were able to defend ourselves, the entire IQ.Lvbs team was banned from the Comma.ai Community Discord server the following morning, on October 8th at 10:42 am. No explanation for this ban was communicated to us.
Having said that, the claim that NovaSpark is Rebellious Hope with a short fine-tune and an unchanged policy does not survive scrutiny, and we’re not sure how Comma came to that conclusion, especially considering that they do not possess NovaSpark’s model weights or ONNX.
Attempts to Steal NovaSpark and Access Users’ Devices
From October 7 to 14, we faced repeated attempts to steal NovaSpark, access users’ devices and private driving files, and exploit our sign-in system. The first requests came from an IP assigned to Comma.ai Inc. Our logs link usernames josh_sp_clark and haraschax to that same IP. Later sessions used konn3kt-device-1.0c, the same client identifier used by josh_sp_clark’s device at Comma’s IP.
They attempted to obtain repository credentials, evade model-access restrictions with falsified device identities and made-up serials, bypass ownership checks to potentially reach users’ devices and private files, and exploit sign-ins through redirects to addresses such as evil.example.com/steal and script injection. They kept changing software identifiers, download URLs, and device IDs after access was denied. They obtained encrypted NovaSpark files, but their requests for the decryption key were denied. We found no evidence that model weights or decryption keys were obtained.
Comma’s IP targets NovaSpark and its decryption key
The first requests came from 67.208.245.10, an IP assigned to Comma.ai Inc. Our account audit also places josh_sp_clark and haraschax at that IP.
Just four minutes after registering, device f9b4c67e4bda0b7c (serial DEV-653886953a4ee144) asked for NovaSpark’s metadata, its encrypted .iqc file, and the key at /assets/iqc-keys/[NOVASPARK_KEY_ID]. It then attempted to access another internal development model named NovaScala. All four requests returned 403 (access denied).
That afternoon, Comma’s IP requested current and older NovaSpark files through file-history and direct-download URLs. It also requested IQ.Pilot’s compiled model-selector modules: git_auth, which handles authentication for model-file requests, and multiple versions of fetcher, which retrieves the model catalog. We believe these binaries were being examined for embedded repository credentials and another way to access NovaSpark’s proprietary weights.
The same IP requested registration.py, cloud_client.py and pairing_dialog.py, plus hephaestusd, manage_hephaestusd, cloud_routes, iqtelemetryd and iquploaderd. These programs handle device registration, account pairing, live connections and data transfers, and describe how a device gets recognized by our services and how it sends and receives files. Cloudflare blocked all requests for the compiled fetcher module at 3:53:27 and 3:53:43 p.m.
Comma’s IP did download IQ.Pilot’s default comma vision and driving policy models (~77 MB) by using the public installation repository’s API with ref=release, which returned 200. The URL was git.konn3kt.com/api/v1/repos/IQ.Lvbs/IQ.Pilot/raw/iqpilot/selfdrive/iqmodeld/default_model/driving_policy_c210m_tinygrad.pkl?ref=release.
Meanwhile, josh_sp_clark’s device 0aa8b488ece5f0ff made 34 requests for the model catalog from Comma’s IP. During those attempts, its User-Agent header (the software identity declared in each request) changed from openpilot-1.0c to konn3kt-device-1.0c, then to konn3kt-device-IQ.Pilot 1.0c. We believe the requester was trying to pass as a client our model service would accept and gain access to NovaSpark. All 34 requests returned 403.
By that evening, requests from Comma’s IP were appearing under iqos-fetch and konn3kt_iq_auth, names used by our internal download accounts. We believe IQ.Pilot’s proprietary binaries were decompiled (violating our license and our Terms of Service) to extract Fernet-encrypted personal access tokens (PATs) for these accounts. The credentials identified in our investigation are iqos-fetch token ID 50 from iqos-image-fetch, used for firmware and maps, and konn3kt_iq_auth token ID 68, used for model repositories. A PAT authenticates repository requests as its account, allowing access to the private files that account can read, but NovaSpark’s decryption key still remained inaccessible.
At 5:35 p.m., another josh_sp_clark device, 7a3086fff1c24499, requested lebrowski_a501760a_amd_policy_oob.pkl through /assets/hf/egpu/oob/ from Comma’s IP. It received a 307 redirect to another download location.
At 5:41 p.m., that device downloaded the 126 MB encrypted NovaSpark file and made two requests for its decryption key. The file request returned 200, but both decryption key requests (required to view the model/steal its weights) returned 403 (denied).
That evening, a separate account, 361 (MyMindBlowingScience), tried malformed registration tokens, setup-claim redemption and a guessed /assets/token URL. The attempts targeted ways to obtain credentials for the model service. Its IPv4 and IPv6 sessions are linked by device 2ec0fbebcf1223d9, and these attempts failed, along with 12 more requests to the model service.
Cross-device access and OAuth exploit attempts
On October 8, we identified about 24 signed-in requests to the repository website under konn3kt_iq_auth, the same internal account name seen in Comma’s IP traffic. Website access would let someone browse the account’s repositories and find private files by name, instead of guessing download URLs.
That evening, M247 VPN address 2a0d:5600:19:41::12 tried changing device IDs in requests for live connections and driving files. The IDs included testdongle, x, a1b2c3d4e5f60718, 0000000000000000 and deadbeefdeadbeef. /v1/route/x/files returned 401, denying access. Other requests sought driving files under d5b9e11f35bc74e4 and the all-zero ID.
As part of this prodding, they also attempted to pull information across different device IDs. This would have allowed a given device to access the private driving data and logs of other devices. Thankfully, our backend’s ownership checks rejected these requests.
The session also tried to open 17 live device connections through WebSocket, using five IDs. All returned 401. The requests used /ws/v2/ with the IDs testdongle, x, 3289dd334ed114f5, 1753ff4a150b039d, and 3a865431577f318c. Several requests used the user agent konn3kt-device-1.0c, also sent by josh_sp_clark’s device from Comma’s IP the day before.
The same session targeted our OAuth sign-in flow, which returns users to our site after authentication. It supplied api-iqlabs.konn3kt.com.evil.example as the return address, with /steal as its destination path. The supplied address began with our hostname, making it look like ours, but the actual domain ended in evil.example. We believe these requests were attempts to capture an OAuth token exchanged during sign-in with an identity provider, such as Google or GitHub, and use it to impersonate a user or gain authenticated access to our services. Other possible aims include exploiting the backend to steal another user’s token or testing for a CORS misconfiguration that could expose protected responses to an outside website.
They also submitted outside destinations through /auth/github, /v2/auth/h/redirect and /v2/auth/, using the redirect, state and redirect_after fields. Variations included //evil.example.com/steal, backslashes in place of normal URL separators, and an outside redirect nested inside a localhost URL. These variations tested whether differences in how the server and browser interpreted the destination could let an outside address past the server’s checks.
Two more inputs were http://localhost:9999/[email protected] and http://localhost:[email protected]/auth (likely probing for vulnerabilities). They also guessed /v2/auth/token, /v2/auth/session, /v2/auth/refresh, /v2/auth/keys and /v2/auth/pubkey. All five returned 404 (not found).
The login request was api-iqlabs.konn3kt.com/auth/github?redirect=https%3A%2F%2Fapi-iqlabs.konn3kt.com.evil.example%2Fsteal.
They also put executable instructions where the login system expected a return address. javascript:alert(1) and data:text/html,<script>alert(1)</script> are small test scripts, which if executed, display a browser alert. These scripts test whether a login input can make the browser run code, and a script running with our site’s permissions could read a signed-in user’s data or maliciously act through their account. All attempts were rejected.
Another input contained encoded line breaks followed by Set-Cookie: x=1, which tries to make the server treat part of the submitted URL as an instruction in its response. If accepted, it could plant a cookie in the user’s browser.
The session also tried to change device records and submit made-up health data. After 47 rejected setup submissions, five successful registration responses produced four IDs: 5c3fa1c4a7bf8051, d5b9e11f35bc74e4, f72ff809e530b54d and c8851085e12d8e96, using 14-digit IMEIs.
It then tried to edit and delete /v1/devices/d5b9e11f35bc74e4 using PATCH and DELETE. These returned 422 and 405 (the submitted content was rejected and deletion was not allowed). An update to f72ff809e530b54d returned 401, and guessed /owner and /state paths returned 404.
Health submissions to /v1/devices/{dongle_id}/rt_health returned 200 for d5b9e11f35bc74e4 and 401 for deadbeefdeadbeef and f72ff809e530b54d. The recorded test values included git_commit:"abc123" and q:"dGVzdA==", which is “test” encoded in base64. A /v1/telemetry/ticket request returned 200, but the report sent to /v1/telemetry/report one second later was denied with 401. Accepting another user’s device ID could let someone change that device’s records, and act on malicious intent. Our ownership checks blocked these attempts before they could change another user’s device records.
Checking repository credentials and permissions
The same M247 address then checked which accounts and files its requests could reach. It alternated between teal and iqos-fetch, queried /api/v1/user and tried the model, firmware and map repositories. iqos-fetch is the internal account name also seen in Comma’s IP traffic.
Combined with the repository requests, we believe this was an attempt to find out what permissions the credentials provided. The requests returned 403 denials and 404 not-found responses.
Changed client headers and more requests for model files
Account 361 (MyMindBlowingScience) returned with more devices and IP addresses. Device 5863432b66cb84dc at 198.44.136.252 downloaded encrypted NovaSpark files, but was denied the key with 403. Another device on that account, 549b6a8c4445de0a, tried the model and key from 173.249.217.38 and its associated IPv6 address, but those requests also returned 403.
Device 5863432b66cb84dc also requested lebrowski_a501760a_amd_policy_oob.pkl, the same separate policy file requested from Comma’s IP on October 7, and received a 307 redirect. We believe the repeated attempts under different IDs were looking for an identity that could obtain both NovaSpark and its key in an attempt to steal the weights.
Another device, 68018d39275e595f at 2a0d:5600:19:39::12, requested model files by their storage-object IDs through the device asset API’s iqmodels-lfs route. It also requested the same catalog with three different User-Agent headers: konn3kt-device-1.0c, also used at Comma’s IP, then curl/8 and konn3kt-device-0.9.9.
We believe it was checking whether a different claimed software version or file-request route would get past the access restrictions. All its model-file requests returned 403. Registration records also show placeholder serials such as serialAAA and serialBBB, along with a separate serial reused across two devices.
New device IDs immediately request model access
Requests under iqos-fetch, the account name also seen at Comma’s IP, tried lebrowski_a501760a_amd_policy.pkl under both teal and IQ.Lvbs. They also tried the made-up repository nosuch/nosuch, apparently to compare its response with real repositories and check whether private repositories existed. These requests returned 404, 401 and 403.
The same M247 session tried to list the contents of IQ.OS, maps and IQ.Pilot through iqos-fetch. It requested /api/v1/repos/iq.lvbs/iqos, /api/v1/repos/IQ.Lvbs/iqmaps and listings across all three repositories, including IQ.Pilot pages 1–17. Those listings would provide file names and paths to request instead of guessing them, as they had been doing up to that point.
Just 5 minutes 24 seconds after device 549b6a8c4445de0a was denied the encrypted NovaSpark file, a new ID, 8d9803b47ca4cf3e, obtained it, but still failed to obtain the decryption key required to steal its weights. Both belonged to account 361 and used 173.249.217.38.
At 1:29 a.m., four more device IDs were registered in 3.273 seconds. Each ID was used to request model access through x.json less than a second after registration, and all four requests were denied with 403.
The submitted serials included abctest, evilexample and admin123, with test IMEIs 350000000000006 and 860000000000009. Our investigation counted more than 34 registrations from M247 and Comma’s IP, including 23 devices with no recorded drives. All non-spoofed devices in this investigation were real comma 4 units. We believe the new identities were being created to find one the model service would accept.
Pairing attempts and 26 authentication-provider names
On October 12, another 14 device IDs appeared. The first four used mixed-case serials and came from 2a0d:5600:19:42::12. We suspect IQ.Pilot’s proprietary registration components were reverse-engineered to reproduce the signed requests used to register devices.
There were also 18 attempts to link devices to accounts: 13 through /v2/pilotpair/qr and five through /v2/pilotpair/manual. All returned 401. Pairing is what gives an account access to a device, so these attempts targeted the step that establishes that permission.
Guesses for /token, /complete and /pair beneath those claims returned 404. We believe these were attempts to turn a setup claim into usable device credentials.
Eight more model requests targeted docs/big_selector_c.json, iqemac_runtime.json, hf/index.json and an iqmodels-lfs object. These sought model lists, runtime configuration and a model file through the asset API’s storage-object route. All eight returned 403. A separate request using the placeholder device ID x was also denied, with 401.
The session tried 26 login-provider names, including anonymous, guest, pat, token and none, using python-requests/2.32.5. All were recorded as denied with 401. These requests tested whether a different provider name would open another way to sign in.
They also supplied evil.com and example.evil.com in the sign-in flow’s state field, again testing outside destinations. The recorded response was 402.
Other requests tried to list and create WireGuard network peers, reach /_admin, pair nodes, open demo sessions, and refresh/exchange login tokens. Creating a network peer could establish another connection to our services, and obtaining a token could allow later requests to authenticate, but their requests were denied. Five requests to /auth/apple/callback also returned 404.
Even more device IDs
Nine more device IDs were registered between 12:59 a.m. and 6:16 p.m., including 4260b2832b834f90, 0cdf01d1f5259b05 and 7140da43673070d5. All were listed without an owner or recorded drives, and further attempts were made to access NovaSpark, all of which were denied.
Comma’s IP comes back for more model requests
On October 14, a week after the first NovaSpark requests, Comma’s IP was still asking for model files. At 8:57 a.m., requests listed under konn3kt_iq_auth asked for model_selector_c.json and other IQModels paths. They all returned 403.
The same day’s findings also record a 200 catalog response for 7a3086fff1c24499, which is the josh_sp_clark device that downloaded the encrypted NovaSpark file from Comma’s IP on October 7. In total, we counted 34 unique Comma 4 device IDs from Comma's IP that were registered to Konn3kt.
October 14 is the last date covered by this report. Our review has found no evidence that encryption keys, decrypted model weights, or user data was obtained. We apologize for the intermittent Konn3kt outages and model availability lapses over the last few days. These interruptions resulted from security updates we deployed in response to the suspicious activity described above. We are evaluating our legal options in response to the activity documented above and will take appropriate steps to protect our systems and intellectual property.
Our New Model!
In more positive news, our next model is out now! We consider IQ.Model Waypoint to be a significant step forward in our model architecture. Its main features are a massive upgrade in navigation capabilities (iterating on the existing navigation trained in NovaSpark), and improved longitudinal behaviors. This is our first model developed with navigation as a major focus, including major changes to the input stack. It even natively accepts text input, such as telling the model to “turn left in 500 feet onto Harney St”. The model also has a native output head for blinker/signal intents, replacing the state machine setup we had previously. Unlike other models, Waypoint commands turn signals from the model itself. We also did a ton of additional RL training to improve stopping at intersections without a lead vehicle. To top it off, we also bumped the temporal context queue of the model from 1.6 seconds to 3 seconds. We hope you enjoy using Waypoint as much as we enjoyed making it. Be sure to stay tuned for all of our model releases by subscribing to the “Models” tag in the IQ.Pilot Discord server.
Made with ❤️ — The IQ.Lvbs team