Add Docker Compose-based setup

[amake]

I am working on an (almost) one-step dev setup powered by Docker Compose.

1. Run docker compose up -d in the repo root
2. Run ./docker_compose_setup.sh (and wait a long time) to do the initial setup

It is almost entirely working, but I am having trouble with:

• Solr search doesn't work. The config and indexing seem to work, but searches always fail with zero results. I note two seemingly related issues:

  • During indexing, there are numerous errors like

    Failed to add documents to Solr: Solr responded with an error (HTTP 400): [Reason: ERROR: [doc=port.port.29001] cannot set an index-time boost, unindexed or norms are omitted for field name: xdaliclock]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/haystack/backends/solr_backend.py", line 100, in update
        self.conn.add(docs, commit=commit, boost=index.get_field_weights())
      File "/usr/local/lib/python3.6/dist-packages/pysolr.py", line 1050, in add
        solrapi=solrapi,
      File "/usr/local/lib/python3.6/dist-packages/pysolr.py", line 565, in _update
        "post", path, message, {"Content-type": "text/xml; charset=utf-8"}
      File "/usr/local/lib/python3.6/dist-packages/pysolr.py", line 463, in _send_request
        raise SolrError(error_message % (resp.status_code, solr_message))
    

    Nevertheless the data and indexes do appear to be created in the Docker volume.

  • If I expose the Solr web console and access the logging section, the log shows the following error repeatedly. The times seem to correspond to when I perform a search in the app.

    org.apache.solr.common.SolrException: can not sort on multivalued field: name_length
        at org.apache.solr.schema.SchemaField.checkSortability(SchemaField.java:164)
        at org.apache.solr.schema.TrieField.getSortField(TrieField.java:163)
        at org.apache.solr.schema.SchemaField.getSortField(SchemaField.java:150)
        at org.apache.solr.search.SortSpecParsing.parseSortSpecImpl(SortSpecParsing.java:202)
        at org.apache.solr.search.SortSpecParsing.parseSortSpec(SortSpecParsing.java:60)
        at org.apache.solr.search.QParser.getSortSpec(QParser.java:285)
        at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:188)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:270)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:724)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:750)
    

    Update: the Solr issues were both due to me configuring the wrong directory with build_solr_schema. After fixing that, I find that search does work correctly.

• There doesn't seem to be any documentation about loading mpstats data, so I haven't managed to set that up yet.

[codecov]

Codecov Report

Merging #339 (514e018) into main (ed3d8ef) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #339   +/-   ##
=======================================
  Coverage   81.17%   81.17%           
=======================================
  Files         111      111           
  Lines        2762     2762           
  Branches      216      216           
=======================================
  Hits         2242     2242           
  Misses        474      474           
  Partials       46       46           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed3d8ef...514e018. Read the comment docs.

[mojca]

Awesome!

mystats data gets filled when users submit their data (a weekly job submits data about installed ports from everyone who installed the mpstats port).

We probably need a script that will generate some fake entries for the sake of development & testing.

[mojca]

Can you please also add some super basic documentation?

[amake]

I found some permissions-related issues and am working on a fix. I'll un-draft when ready.

[arjunsalyan]

Thanks, there is some sample data for mpstats here - we use it for performing tests. https://github.com/macports/macports-webapp/blob/main/app/tests/sample_data/submissions.json

But the data needs to modified, at least the port names.

[amake]

This is now ready. I had to adjust the data storage strategy in order to get things working seamlessly on first run. Previously I had accidentally hit upon a strategy that only works if you manually adjust things as you go.

Thanks, there is some sample data for mpstats here

Thanks. If people don't mind I'd like to skip mpstats for this PR.