A product URL returns HTTP 200, but the browser displays “Product not found.” In a JavaScript storefront, the successful response may contain only an application shell. A later data request determines whether the product exists and what the visitor eventually sees.
Diagnose three states separately: the document response, the product API result and the rendered content. A status-only check can count an empty or broken product page as a healthy request because the first response was successfully transmitted.
Google's HTTP status documentation describes successful responses containing empty or error-like content as potential soft 404s. HTTP 200 does not guarantee indexing. The correction depends on why the page is missing, though. A deleted product and a temporary database outage should not receive the same treatment.
Establish whether the product should exist
Start with one affected URL and check its ID or slug in the authoritative product inventory. A randomly guessed address and a product that is still offered belong in different investigation groups. Otherwise, the team can “repair” a correct missing-page response while overlooking a real availability failure.
Record where the URL came from as well. Is a category still linking to it? Is an active campaign using it? Is it an old route from a migration? This context helps prioritize the issue and prevents a technical change from accidentally restoring content deliberately removed from the store.
Indexing is not determined by transport success alone. A server can successfully transmit a document that contains none of the promised product information. Conversely, a genuine 404 for an address with no corresponding product may be the correct result, not an SEO defect to remove.
Before assigning work, write down the expected state. “This product exists and must show its details” gives a developer a different target from “this retired URL should remain unavailable.” A report containing only a red status in a crawler leaves that decision unresolved.
A fictional store with two different failures
This example is invented. A store at https://example.com/products/lamp-72 first serves its navigation, footer and an empty product area. The document response has status 200. The application then requests https://example.com/api/products/lamp-72.
In the first case, the API returns 404 because that product ID does not exist. The browser displays a not-found message. The original document response remains the successful response already sent. An HTTP check that does not render the application initially sees no failure.
In the second case, the lamp still exists, but the API returns 503 or times out. An overly broad error handler displays the same not-found message. The visitor sees an identical screen, yet the application has not established that the product is missing. It currently cannot determine the product's state.
A handler that converts every exception into an empty product list or missing-page view hides this distinction. Keep at least three outcomes separate: found, confirmed missing and temporarily unavailable. Preserve them in diagnostic records too, rather than storing every failed lookup under a single “not found” label.
Inspect the document before the data request
Reload the address directly and select the Document request in the browser's network panel. Record the status, final URL and any redirects. Save this response before investigating the product data request so the two are not confused later.
An independent HTTP fetch can save the headers and body:
bashcurl -sS -D document-headers.txt -o document.html "https://example.com/products/lamp-72"
On Windows PowerShell, use curl.exe if needed. This command does not execute JavaScript. It therefore cannot establish whether product data arrives after the initial load. Search the saved body for a concrete product detail and distinguish actual content from an empty loading placeholder.
Return to the browser and inspect the related data request. Capture its status, response content and timing. An API may also return HTTP 200 with an application-level failure in its JSON, so the status number alone is insufficient at this layer too.
Test the public page without relying on an existing login or a previously prepared shopping session. Session state can affect the response. Record these conditions instead of generalizing an authenticated success to ordinary visitors or crawlers. After a website migration, check whether old and new product routes reach the intended data source.
Keep the response files associated with the time and release tested. A later successful request does not show what the earlier failed response contained. If the problem is intermittent, matching the page request to its data request and release is often more useful than collecting screenshots without timestamps.
Compare direct loading with client-side navigation
Open the same product through an internal category link. A client-side transition may request data or a framework payload without requesting a new complete HTML document. A 404 from the API is still not the status of a newly fetched product document.
Now reload the destination directly. If direct loading fails while internal navigation works, the server may resolve the route differently from the browser. If only the direct load succeeds, investigate the client router, its cache and its error state. This narrows the search without proving a particular line of code is responsible.
Also navigate from a missing product to an existing one. A stale error view or noindex instruction can affect the next valid page if the application does not reset it. Repeat that transition with back and forward navigation to catch restored state.
For crawling, a relevant URL needs to work independently of a previously successful session. A click path that succeeds only because the browser already holds product data is not sufficient acceptance evidence.
If the router changes the visible address while retaining old product content during loading, record that intermediate state as well. The final state may be correct, but a failed data request could leave the old content under the new URL. Do not mistake a previous product's details for a successfully loaded destination.
Include temporary failure in the test matrix
Reproduce the required states in an appropriate test environment. Do not delete real products or interrupt a production API to manufacture a failure. Controlled response cases and a documented test dataset provide a safer, more reproducible check.
| Case | Data state | Evidence to inspect |
|---|---|---|
| Existing product loaded directly | Valid product response | Success status and actual product content |
| Nonexistent product ID loaded directly | Confirmed missing | Intended not-found handling |
| Existing product during API failure | Temporarily unknown | No false claim that the product was deleted |
| Missing product through an internal link | Confirmed missing | Router, error view and indexing instruction |
| Transition from missing to valid product | Valid again | No leftover error or noindex state |
| Empty category | Intentionally empty or data failure | Correct distinction rather than an automatic 404 |
Write the expected behavior for each row before running it. Record the document status, API outcome, visible primary content and effective robots instructions. A screenshot alone cannot supply those fields. This bundle lets a developer determine whether the failure originates in state classification or only in presentation.
Handle the state where it becomes known
If the server can establish that the product does not exist before sending the document, it should express that state appropriately. Google recommends meaningful HTTP statuses. For fully client-rendered applications, its JavaScript SEO guidance describes alternatives when an appropriate document status is impractical: redirect to a URL returning a real 404, or add noindex to the error page with JavaScript.
Those alternatives need targeted checks. A redirect destination must actually return the intended error status. A dynamically inserted noindex must not survive a later transition to a valid product. A single responsible metadata layer is easier to inspect than several components independently appending robots tags.
Streaming adds a timing constraint. The current Next.js not-found documentation describes a 404 before streaming starts, while a not-found state after streaming begins retains 200 and includes noindex. Check the version actually deployed. Rendering a not-found component does not by itself establish the response's HTTP status.
Do not therefore reduce every delayed API failure to a copy change in the visible error message. Establish whether the framework can determine product existence earlier and which indexing controls apply to the chosen delivery path. Text rendered later in the browser cannot replace headers that have already been sent.
Inspect the control on a fresh direct request too. Fixing only a client-side transition may leave the server response unchanged. Conversely, a corrected server path may still be followed by a client component that restores an old error instruction. Acceptance needs both paths from the original matrix.
A timeout is not a permanent deletion
When product data is temporarily unavailable, “this product does not exist” is unsupported. A blanket noindex on every timeout can target valid products. A permanent redirect created only because of a short-lived backend failure makes the same category mistake.
Model the outage as its own operational state. If the server cannot supply the requested page because of a temporary failure, the response needs appropriate error handling. If the site serves a useful cached product page instead, check which facts remain reliable. A passing SEO test is no reason to invent a current price or availability.
An empty category also requires a deliberate decision. Google's ecommerce URL guidance recommends noindex where a category has no useful items and identifies 404 as an option when an empty category is removed from search and browsing. An API failure incorrectly producing zero products does not establish that state.
A canonical tag cannot make this state decision for you. Canonicalizing every broken product page to its category neither restores the missing data nor turns an error view into an equivalent copy of the category content.
Check for signals left behind by the error
Repeat the same matrix cases in the same environment after the repair. Confirm content on valid products, intended handling on confirmed missing products and a separate state for temporary outages. Compare direct loads with browser navigation so the correction covers both routes.
The Crawl Foundry site audit workflow helps relate technical findings to a clear URL scope. Add saved document responses and the corresponding product data outcomes for a soft-404 investigation. A status-code export alone cannot fully describe an error that appears later in rendered content.
Removed product URLs should not remain in the XML sitemap as pages intended for search. Check internal references too. A well-designed missing-page view does not make an accidentally deleted, prominently linked product correct.
Use Google Search Console to observe the affected classification and processing date. A current live test and an older indexed observation may temporarily describe different states. That is not evidence that the same unchanged fix needs to be deployed repeatedly.
If a valid page instead remains crawled but currently not indexed, investigate that status separately. The technical result here is specific: existing products deliver their content, confirmed missing products receive the intended handling, and a temporary data failure is no longer presented as a permanent deletion.