Skip to content

Commit 6c67c61

Browse files
Dr-IrvDr-Irv
authored
Add notebook and data from July 2024 webinar (#81)
1 parent 722ea80 commit 6c67c61

File tree

3 files changed

+384
-0
lines changed

3 files changed

+384
-0
lines changed

‎webinar/2407-data-first/mcfdata.v1.xlsx

6.87 KB
Binary file not shown.

‎webinar/2407-data-first/mcfdata.v2.xlsx

11.4 KB
Binary file not shown.

‎webinar/2407-data-first/netflow.ipynb

+384
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,384 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Multicommodity Flow Example Using `gurobipy-pandas`\n",
8+
"\n",
9+
"#### Author: Irv Lustig, Optimization Principal, Princeton Consultants\n",
10+
"\n",
11+
"Solve a multi-commodity flow problem. There are multiple products, which can be \n",
12+
"produced in multiple locations, and have to be shipped over a network to other locations.\n",
13+
"Each location may have supply and/or demand for any product. The network may have\n",
14+
"transhipment locations where freight is interchanged. For each arc in the network, there is \n",
15+
"a limited capacity of the total products that can be carried. Each arc also has a product-specific\n",
16+
"cost for shipping one unit of the product on that arc.\n",
17+
"\n",
18+
"This example is based on `netflow.py` that is supplied by Gurobi.\n"
19+
]
20+
},
21+
{
22+
"cell_type": "markdown",
23+
"metadata": {},
24+
"source": [
25+
"#### Import necessary libraries\n",
26+
"\n",
27+
"- `IPython.display` is used to improve the display of `pandas` `Series` by converting them to `DataFrame` for output\n",
28+
"- `PyQt5.QtWidgets` allows prompting for a data file\n"
29+
]
30+
},
31+
{
32+
"cell_type": "code",
33+
"execution_count": null,
34+
"metadata": {
35+
"slideshow": {
36+
"slide_type": "slide"
37+
}
38+
},
39+
"outputs": [],
40+
"source": [
41+
"from IPython.display import display\n",
42+
"import pandas as pd\n",
43+
"import gurobipy as grb\n",
44+
"import gurobipy_pandas as gppd"
45+
]
46+
},
47+
{
48+
"cell_type": "code",
49+
"execution_count": null,
50+
"metadata": {},
51+
"outputs": [],
52+
"source": [
53+
"%gui qt\n",
54+
"\n",
55+
"from PyQt5.QtWidgets import QFileDialog\n",
56+
"\n",
57+
"def gui_fname(dir=None):\n",
58+
" \"\"\"Select a file via a dialog and return the file name.\"\"\"\n",
59+
" if dir is None: dir ='./'\n",
60+
" fname = QFileDialog.getOpenFileName(None, \"Select data file...\", \n",
61+
" dir, filter=\"All files (*);; SM Files (*.sm)\")\n",
62+
" return fname[0]"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {},
68+
"source": [
69+
"#### Get the file from a prompt"
70+
]
71+
},
72+
{
73+
"cell_type": "code",
74+
"execution_count": null,
75+
"metadata": {},
76+
"outputs": [],
77+
"source": [
78+
"filename = gui_fname()"
79+
]
80+
},
81+
{
82+
"cell_type": "markdown",
83+
"metadata": {},
84+
"source": [
85+
"#### Read Data using `pandas`\n",
86+
"\n",
87+
"Read in the data from an Excel file. Converts the data into a dictionary of `pandas` `Series`, with the assumption that the last column is the data column."
88+
]
89+
},
90+
{
91+
"cell_type": "code",
92+
"execution_count": null,
93+
"metadata": {
94+
"slideshow": {
95+
"slide_type": "slide"
96+
}
97+
},
98+
"outputs": [],
99+
"source": [
100+
"raw_data = pd.read_excel(filename, sheet_name=None)\n",
101+
"data = {\n",
102+
" k: df.set_index(df.columns[:-1].to_list())[df.columns[-1]]\n",
103+
" for k, df in raw_data.items()\n",
104+
"}\n",
105+
"for k, v in data.items():\n",
106+
" print(k)\n",
107+
" display(v.to_frame())"
108+
]
109+
},
110+
{
111+
"cell_type": "markdown",
112+
"metadata": {},
113+
"source": [
114+
"# Data Model\n",
115+
"\n",
116+
"## Sets\n",
117+
"\n",
118+
"| Notation | Meaning | Table Locations |\n",
119+
"| ---- | --------------------------- | ----------- | \n",
120+
"| $\\mathcal N$ | Set of network nodes | `cost`: Columns `From`, `To` <br> `capacity`: Columns `From`, `To` <br> `supply`: Column `Node` <br> `demand`: Column `Node` |\n",
121+
"| $\\mathcal P$ | Set of products (commodities) | `cost`: Column `Product` <br> `supply`: Column `Product` <br> `demand`: Column `Product` |\n",
122+
"| $\\mathcal A$ | Set of arcs $(n_f,n_t)$, $n_f,n_t\\in\\mathcal A | `cost`: Columns `From`, `To` |\n",
123+
"| $\\mathcal P_a$ | Set of products $p\\in\\mathcal P$ that can be carried on arc $a\\in\\mathcal A$ | `cost`: Columns `Product`, `From`, `To` |\n",
124+
"| $\\mathcal A_p$ | Set of arcs $a\\in\\mathcal A$ that can carry product $p\\in\\mathcal P$ | `cost`: Columns `Product`, `From`, `To` |\n",
125+
"\n",
126+
"\n",
127+
"\n",
128+
"## Numerical Input Values\n",
129+
"\n",
130+
"The input data is converted to pandas `Series`, so the name of each `Series` is also the name of the value.\n",
131+
"\n",
132+
"| Notation | Meaning | Table Name/Value Column | Index Columns \n",
133+
"| ---- | --------------------------- | ------ | ---------- |\n",
134+
"| $\\kappa_a$ | Capacity of arc $a\\in\\mathcal A$ | `capacity` | `From`, `To` |\n",
135+
"| $\\pi_{ap}$ | Cost of carrying product $p$ on arc $a\\in\\mathcal A$, $p\\in\\mathcal P_a$, | `cost` | `Product`, `From`, `To` |\n",
136+
"| $\\sigma_{pn}$ | Supply of product $p\\in\\mathcal P$ at node $n\\in\\mathcal N$. Defaults to 0 | `supply` | `Node` |\n",
137+
"| $\\delta_{pn}$ | Demand of product $p\\in\\mathcal P$ at node $n\\in\\mathcal N$. Defaults to 0 | `demand` | `Node` |"
138+
]
139+
},
140+
{
141+
"cell_type": "markdown",
142+
"metadata": {},
143+
"source": [
144+
"## Compute Sets\n",
145+
"\n",
146+
"- The set $\\mathcal P$ of products can appear in any of the tables `supply`, `demand` and `cost` .\n",
147+
"- The set $\\mathcal N$ of nodes can appear in any of the tables `capacity`, `supply`, `demand` and `cost`"
148+
]
149+
},
150+
{
151+
"cell_type": "code",
152+
"execution_count": null,
153+
"metadata": {},
154+
"outputs": [],
155+
"source": [
156+
"commodities = set(\n",
157+
" pd.concat(\n",
158+
" [\n",
159+
" data[dfname].index.to_frame()[\"Product\"]\n",
160+
" for dfname in [\"supply\", \"demand\", \"cost\"]\n",
161+
" ]\n",
162+
" ).unique()\n",
163+
")\n",
164+
"commodities"
165+
]
166+
},
167+
{
168+
"cell_type": "code",
169+
"execution_count": null,
170+
"metadata": {},
171+
"outputs": [],
172+
"source": [
173+
"nodes = set(\n",
174+
" pd.concat(\n",
175+
" [\n",
176+
" data[dfname].index.to_frame()[fromto].rename(\"Node\")\n",
177+
" for dfname in [\"capacity\", \"cost\"]\n",
178+
" for fromto in [\"From\", \"To\"]\n",
179+
" ]\n",
180+
" + [data[dfname].index.to_frame()[\"Node\"] for dfname in [\"supply\", \"demand\"]]\n",
181+
" ).unique()\n",
182+
")\n",
183+
"\n",
184+
"nodes"
185+
]
186+
},
187+
{
188+
"cell_type": "markdown",
189+
"metadata": {},
190+
"source": [
191+
"### Compute the Net Flow for each node\n",
192+
"\n",
193+
"The net flow $\\mu_{pn}$ for each product $p\\in\\mathcal P$ and node $n\\in\\mathcal N$ is the sum of the supply less the demand. For transshipment nodes, this value is 0. This is called `inflow` in the code."
194+
]
195+
},
196+
{
197+
"cell_type": "code",
198+
"execution_count": null,
199+
"metadata": {},
200+
"outputs": [],
201+
"source": [
202+
"inflow = pd.concat(\n",
203+
" [\n",
204+
" data[\"supply\"].rename(\"net\"),\n",
205+
" data[\"demand\"].rename(\"net\") * -1,\n",
206+
" pd.Series(\n",
207+
" 0,\n",
208+
" index=pd.MultiIndex.from_product(\n",
209+
" [commodities, nodes], names=[\"Product\", \"Node\"]\n",
210+
" ),\n",
211+
" name=\"net\",\n",
212+
" ),\n",
213+
" ]\n",
214+
").groupby([\"Product\", \"Node\"]).sum()\n",
215+
"inflow"
216+
]
217+
},
218+
{
219+
"cell_type": "markdown",
220+
"metadata": {},
221+
"source": [
222+
"## Create the Gurobi Model"
223+
]
224+
},
225+
{
226+
"cell_type": "code",
227+
"execution_count": null,
228+
"metadata": {},
229+
"outputs": [],
230+
"source": [
231+
"m = grb.Model(\"netflow\")"
232+
]
233+
},
234+
{
235+
"cell_type": "markdown",
236+
"metadata": {},
237+
"source": [
238+
"## Model\n",
239+
"\n",
240+
"### Decision Variables\n",
241+
"\n",
242+
"The model will have one set of decision variables:\n",
243+
"- $X_{pa}$ for $p\\in\\mathcal P$, $a\\in\\mathcal A_p$ represents the amount shipped of product $p$ on arc $a$. We will call this variable `flow` in the code.\n",
244+
"\n",
245+
"The cost of shipment is $\\pi_{ap}$. \n",
246+
"\n",
247+
"This defines the objective function:\n",
248+
"$$\n",
249+
"\\text{minimize}\\quad\\sum_{a\\in\\mathcal A}\\sum_{p\\in\\mathcal P_a} \\pi_{ap}X_{pa}\n",
250+
"$$"
251+
]
252+
},
253+
{
254+
"cell_type": "code",
255+
"execution_count": null,
256+
"metadata": {},
257+
"outputs": [],
258+
"source": [
259+
"flow = gppd.add_vars(m, data[\"cost\"], obj=data[\"cost\"], name=\"flow\")\n",
260+
"m.update()\n",
261+
"flow"
262+
]
263+
},
264+
{
265+
"cell_type": "markdown",
266+
"metadata": {},
267+
"source": [
268+
"### Constraints\n",
269+
"\n",
270+
"#### Flow on each arc is capacitated\n",
271+
"\n",
272+
"$$\n",
273+
"\\sum_{p\\in\\mathcal P_a} X_{pa} \\le \\kappa_a\\qquad\\forall a\\in\\mathcal A\n",
274+
"$$"
275+
]
276+
},
277+
{
278+
"cell_type": "code",
279+
"execution_count": null,
280+
"metadata": {},
281+
"outputs": [],
282+
"source": [
283+
"capct = pd.concat(\n",
284+
" [flow.groupby([\"From\", \"To\"]).agg(grb.quicksum), data[\"capacity\"]], axis=1\n",
285+
").gppd.add_constrs(m, \"flow <= capacity\", name=\"cap\")\n",
286+
"m.update()\n",
287+
"capct"
288+
]
289+
},
290+
{
291+
"cell_type": "markdown",
292+
"metadata": {},
293+
"source": [
294+
"#### Conservation of Flow\n",
295+
"\n",
296+
"For each node and each product, the flow out of the node, less the flow into the node is equal to the net flow.\n",
297+
"\n",
298+
"$$\n",
299+
"\\sum_{(n, n_t)\\in A_p} X_{p(n,n_t)} - \\sum_{(n_f, n)} X_{p(n_f,n)} = \\mu_{pn}\\qquad\\forall p\\in\\mathcal P, n\\in\\mathcal N\n",
300+
"$$"
301+
]
302+
},
303+
{
304+
"cell_type": "code",
305+
"execution_count": null,
306+
"metadata": {},
307+
"outputs": [],
308+
"source": [
309+
"flowct = pd.concat(\n",
310+
" [\n",
311+
" flow.rename_axis(index={\"From\": \"Node\"})\n",
312+
" .groupby([\"Product\", \"Node\"])\n",
313+
" .agg(grb.quicksum)\n",
314+
" .rename(\"flowout\"),\n",
315+
" flow.rename_axis(index={\"To\": \"Node\"})\n",
316+
" .groupby([\"Product\", \"Node\"])\n",
317+
" .agg(grb.quicksum)\n",
318+
" .rename(\"flowin\"),\n",
319+
" inflow,\n",
320+
" ],\n",
321+
" axis=1,\n",
322+
").fillna(0).gppd.add_constrs(m, \"flowout - flowin == net\", name=\"node\")\n",
323+
"m.update()\n",
324+
"flowct"
325+
]
326+
},
327+
{
328+
"cell_type": "markdown",
329+
"metadata": {},
330+
"source": [
331+
"# Optimize!"
332+
]
333+
},
334+
{
335+
"cell_type": "code",
336+
"execution_count": null,
337+
"metadata": {},
338+
"outputs": [],
339+
"source": [
340+
"m.optimize()"
341+
]
342+
},
343+
{
344+
"cell_type": "markdown",
345+
"metadata": {},
346+
"source": [
347+
"# Get the Solution\n",
348+
"\n",
349+
"Only print out arcs with flow, using pandas"
350+
]
351+
},
352+
{
353+
"cell_type": "code",
354+
"execution_count": null,
355+
"metadata": {},
356+
"outputs": [],
357+
"source": [
358+
"soln = flow.gppd.X\n",
359+
"soln.to_frame().query(\"flow > 0\").sort_index()"
360+
]
361+
}
362+
],
363+
"metadata": {
364+
"kernelspec": {
365+
"display_name": "gurobi1100py311",
366+
"language": "python",
367+
"name": "python3"
368+
},
369+
"language_info": {
370+
"codemirror_mode": {
371+
"name": "ipython",
372+
"version": 3
373+
},
374+
"file_extension": ".py",
375+
"mimetype": "text/x-python",
376+
"name": "python",
377+
"nbconvert_exporter": "python",
378+
"pygments_lexer": "ipython3",
379+
"version": "3.11.3"
380+
}
381+
},
382+
"nbformat": 4,
383+
"nbformat_minor": 2
384+
}

0 commit comments

Comments
 (0)