Skip to main content
Version: v2.3.4

Ticker Search & Company Data Issues

Overview​

The AlphaSense Enterprise Insight platform in Private Cloud provides ticker search and autocomplete functionality through the company-universe service and a ticker database backed by OpenSearch.

Ticker search supports lookup by ticker symbol, ISIN, and company name, and feeds into document search, company pages, and logo display. Because company-universe runs as a single pod with no auto-scaling by default, it is prone to becoming a bottleneck under high-traffic conditions. Ticker data accuracy also depends on a migration cron job that synchronises ticker data into OpenSearch.

This document serves to outline common failure scenarios related to ticker search and company data and provide troubleshooting steps for resolution.

Failure Scenarios​

1. Ticker Autocomplete Returns No Results or Times Out​

Triage:​

Users report that typing a ticker symbol produces no dropdown results, or the dropdown is extremely slow to appear.

This is typically caused by the company-universe pod being overwhelmed under load — as it runs as a single pod with no auto-scaling — or by the ticker database being saturated by long-running queries such as migration jobs consuming connections and resources.

Troubleshooting:​

Test ticker autocomplete via API to confirm the failure:

curl "https://<pc-domain>/api/company/autocomplete?query=AAPL" \
  -H "Authorization: Bearer <token>"

Expected response time is within 1–2 seconds. A timeout or empty response confirms the issue.

Check the ticker service related pod status and resource usage (Alphasense):

kubectl get pods -n applications | grep ticker
kubectl get pods -n platform | grep company-universe
kubectl get pods -n as-elasticsearch | grep es-ticker-es-data

Look for high CPU/memory usage or only a single pod running. If the pod is under pressure, scale it up immediately.

Check for long-running queries:

kubectl exec -n as-elasticsearch es-ticker-es-<pod> -- mysql

# Find and identify long-running queries
SHOW FULL PROCESSLIST;

# Kill a specific blocking query using its ID
KILL <query-id>;

Also be aware of benign Elasticsearch log noise during index/alias switches — some 403/index-not-found style errors can appear transiently when indices are swapped. Validate impact by confirming whether the autocomplete APIs are actually failing for users.

Caution: Do not kill queries without understanding their impact. Confirm with the team before terminating migration or data sync jobs.

2. ISIN Search Not Working / Ticker Data Out of Sync​

Triage:​

Users report that searching by ISIN returns no results, or that a ticker/ISIN mapping is correct in SaaS but incorrect or stale in the Private Cloud environment.

Common underlying causes:

  • Ticker app versions were upgraded but the Elasticsearch cluster was not upgraded to a compatible version, causing downstream migration/sync incompatibilities.
  • The ticker migration cron job is not running (or is failing) so changes since a cutover date never land in the ticker index.
  • Partial/inconsistent ticker documents in Elasticsearch (wrong asId / all_as_ids / useful_as_ids) for specific tickers.

Troubleshooting:​

  1. Confirm the symptom — test ISIN search directly:

    curl "https://<pc-domain>/api/company/search?isin=<ISIN>" \
      -H "Authorization: Bearer <token>"

    Capture the specific ISIN(s) and the incorrect company mapping shown.

  2. Check ticker migration cron/job health (Alphasense):

    • Identify the migration/sync cron job(s) responsible for applying ticker/ISIN updates.
    • Verify the job is scheduled and succeeding (recent runs, exit codes, error logs).
    • If the job is not running, fix scheduling/config and rerun the migration.

  3. Validate the mapping in Elasticsearch:

    • Exec into the Elasticsearch pod and query the ticker index to confirm what document is stored for the affected ISIN / ticker.
    • If the mapping is wrong only for a small number of tickers, compare with SaaS prod and determine whether it's safe to patch the document as a short-term mitigation.

Mitigation / Remediation (Alphasense):​

  • If cluster/app incompatibility is detected: upgrade Elasticsearch to the expected version, then rerun the ticker migration cron.
  • If migration cron is failing: fix the job and rerun/backfill.
  • If specific ticker docs are malformed: patching the Elasticsearch document can be used as a temporary mitigation (see Failure Scenario 5).

3. Elasticsearch Query Index Growth / Resource Exhaustion​

Triage:​

Symptoms may present as a broad "company data / ticker-related" degradation (slow or failing company pages, sluggish search-related APIs, intermittent timeouts), but the underlying issue is the Elasticsearch cluster used to store search history/query artifacts growing rapidly and exhausting resources. In some cases, rapid query growth can be traced to automated clients executing saved searches at high volume.

Typical signals:

  • Sudden growth in number of stored queries (orders of magnitude above human traffic)
  • Elasticsearch CPU / memory pressure, shard issues, or index size growth accelerating
  • Platform-wide downstream symptoms: elevated latency/timeouts in GraphQL/search APIs and related experiences

Common cause pattern:​

An automated "daily digest"/ingestion workflow re-executes saved searches (e.g., via GraphQL searchById / saved-search replay). Those service-generated searches may be written into Elasticsearch alongside real user queries, causing rapid index growth.

Troubleshooting:​

  1. Confirm Elasticsearch pressure / query growth:

    kubectl get pods -n as-elasticsearch | egrep -i "es-"

    Check Kubernetes resource usage:

    # Requires metrics-server
    kubectl top pods -n as-elasticsearch | egrep -i "es-"
    kubectl top nodes

    # Describe pods for restarts/evictions/OOMs
    kubectl describe pod -n as-elasticsearch <es_pod_name>

    # Recent logs (look for GC pressure, circuit breakers, shard failures)
    kubectl logs -n as-elasticsearch <es_pod_name> --since=30m

    Look for abnormal ingestion rates or large increases in the number of stored queries over short time windows.

  2. Identify the caller / traffic source:

    • Determine whether the high-volume traffic is coming from human UI (webapp) vs automated clients.
    • If available in logs/telemetry, validate clientId (e.g., webapp) and/or the calling service identity.

  3. Validate the API path and behavior:

    • In some environments, automated clients may call GraphQL-search and route to doc-search-realtime (per DOC_SEARCH_HOST configuration in graphql-search).
    • Daily-digest style flows often use searchById (saved search replay). Note: searchById does not expose an input flag to control search-history saving directly; behavior depends on graphql-search/doc-search modes.
    • If investigating whether search history is being saved, capture the outbound doc-search payload/logs and confirm the presence (or absence) of a mode like SEARCHHISTORY_SKIP.

  4. Check for version mismatches / regressions (graphql-search):

    • In some investigations, teams have found that newer PC environments include outbound doc-search modes with SEARCHHISTORY_SKIP by default.
    • Compare graphql-search image/version between the affected private cloud environment and a known-good PC environment; reproduce in an environment with the same image/version when possible.

Mitigation / Remediation:​

  • Short-term (operational): Purge/clear old query data in Elasticsearch to immediately relieve pressure (e.g., remove data older than 1 year), then monitor ingestion rate.
  • Medium-term (product/infra): Filter ingested queries so only genuine human UI traffic is stored (e.g., filter by clientId=webapp) to exclude automated/service traffic.
  • Client-side fix (preferred when possible): Ensure internal services do not use realtime/main doc-search variants that write history by default, or explicitly set parameters (e.g., shouldSaveSearchHistory=false) when executing automated searches.
  • Platform-side safeguard: Consider enforcing SEARCHHISTORY_SKIP (or equivalent) server-side where appropriate to prevent accidental index pollution by non-webapp clients.
  • Version alignment: Upgrade graphql-search in the affected environment to the known-good release lineage where modes are correctly populated.

4. Keyword Suggestions Degraded / "CancellationException" Errors​

Triage:​

Users report keyword suggestions/autocomplete (suggestion-ws) intermittently failing or returning no results. Teams may observe CancellationException (or similar) in logs.

Prior investigations have referenced repeated suggestion-ws failures and CancellationException logs; correlate with overall search-service health and any concurrent Elasticsearch pressure (see Failure Scenario 3).

Troubleshooting:​

  • Confirm whether the issue reproduces only in PC environments.
  • Collect representative timestamps and error logs from suggestion-ws and upstream dependencies (graphql-search / doc-search-realtime).
  • If occurring alongside Failure Scenario 3, check Elasticsearch pressure and overall search-service health first.

Next steps:​

Coordinate with AlphaSense Support (see how to contact AlphaSense Support) to confirm whether the Private Cloud environment is seeing the same CancellationException pattern and to identify the specific upstream request path causing cancellations.

5. Missing Documents / Wrong Company Resolution for Specific Tickers​

Triage:​

A small set of tickers behave incorrectly (e.g., missing docs for a specific ticker, wrong company resolved, or the ticker exists but points to the wrong asId) while most tickers behave normally.

Troubleshooting:​

If you can safely validate the correct values from SaaS prod (or a known-good environment), a temporary mitigation is to patch the affected ticker document in Elasticsearch.

General approach:

  1. Identify the ticker document in Elasticsearch (by countryTicker.raw, all_as_ids, etc.).
  2. Update the document fields to match known-good values (asId, other_as_ids, all_as_ids, useful_as_ids).
  3. Re-query to confirm the update, then re-test the user flow.

Query by all_as_ids:

curl -X GET "http://localhost:9200/ticker-new/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {"match": {"all_as_ids": "<ASID>"}}
  }'

Update by document _id:

curl -X POST "http://localhost:9200/ticker-new/_update/<doc_id>" \
  -H 'Content-Type: application/json' \
  -u "admin:admin" \
  -d '{
    "doc": {
      "asId": "<correct_asid>",
      "other_as_ids": ["..."],
      "all_as_ids": ["..."],
      "useful_as_ids": ["..."]
    }
  }'

Caution: Treat this as a tactical mitigation only — it can be overwritten by later migrations. Prefer fixing the underlying pipeline/migration issue when the problem is broad (see Failure Scenario 2).

6. Company Logos Missing​

Triage:​

Ticker search returns results and company pages load, but company logos are missing/broken (often rendering as empty/placeholder), sometimes for all tickers.

Common patterns seen in prior occurrences:

  • graphql-company fails to load company data/logo from company-universe and returns null when company-universe returns 500.
  • graphql-company-feature-data encounters Redis DNS resolution issues (e.g., getaddrinfo ENOTFOUND redis-graphql-company-feature-data), which can indirectly break company-feature enrichment paths used by the UI.
  • Upstream logo ingestion failures (e.g., Crunchbase logo URL changes) can cause logos to be missing even when runtime services are healthy.

Troubleshooting:​

  1. Determine scope:

    • Is it isolated to a subset of companies/tickers, or are all logos missing?
    • Does it reproduce across environments?

  2. Check runtime service health and errors (Alphasense):

    • Check company-universe health and logs for 500 responses.
    • Check graphql-company logs for company-universe failures (500s) and confirm whether the logo fields are returning null.
    • Check graphql-company-feature-data logs for Redis-related errors, especially DNS resolution failures like getaddrinfo ENOTFOUND redis-graphql-company-feature-data.
    • If DNS errors are present: verify the Redis service name, endpoints, and cluster DNS/CoreDNS health in the namespace.

  3. Validate logo endpoint behavior (API-level):

    • Validate the company logo endpoint used by graphql-company (often of the form /companies/<companyId>/logo) returns a non-empty image/redirect and a correct status.
    • If the endpoint errors or returns empty, correlate with company-universe logs and upstream dependency errors.

  4. Distinguish runtime vs data/ingestion problems:

    • If services are healthy but logos are missing for many companies, investigate upstream ingestion failures (e.g., Crunchbase logo fetch failures / changed base URLs) and verify that the expected logo URL is present in the company profile data.

Mitigation / Remediation:​

  • If company-universe is returning 500: mitigate as a service incident (scale/restart if appropriate, investigate underlying dependency failures).
  • If Redis DNS resolution is failing: fix the DNS/service discovery issue (service name, endpoints, CoreDNS, network policy) and restart affected pods if needed.
  • If upstream ingestion is the issue (e.g., Crunchbase URL changes): coordinate with the owning data pipeline team to update the fetch logic and backfill.

If this occurs in a customer environment, please reach out to AlphaSense Support for coordinated investigation/ownership.