'LinAlgError: Singular matrix' when running lcm_estimate

[lisalan520]

Hi there,

First off, I'm still at location choice model (I got crash when I tried to update to discrete choice model). I get the following error when I run "elcm_choice" model with my own data. Do you have any idea what is going on? Thanks very much!

Here is the message in ipython notebook:

Running model 'elcm_estimate'

LinAlgError Traceback (most recent call last)
in ()
----> 1 sim.run(["elcm_estimate"])

C:\Anaconda\lib\site-packages\urbansim\sim\simulation.pyc in run(models, years, data_out, out_interval)
1458 model = get_model(model_name)
1459 t2 = time.time()
-> 1460 model()
1461 print("Time to execute model '{}': {:.2f}s".format(
1462 model_name, time.time()-t2))

C:\Anaconda\lib\site-packages\urbansim\sim\simulation.pyc in call(self)
670 kwargs = _collect_variables(names=self._argspec.args,
671 expressions=self._argspec.defaults)
--> 672 return self._func(**kwargs)
673
674 def _tables_used(self):

C:\Users\xzhang\Documents\PythonScripts\Marion_urbansim_test_0514_with_building_ids\models.pyc in elcm_estimate(jobs, buildings, zones)
45 def elcm_estimate(jobs, buildings, zones):
46 return utils.lcm_estimate("elcm.yaml", jobs, "building_id",
---> 47 buildings, zones)
48
49

C:\Users\xzhang\Documents\PythonScripts\Marion_urbansim_test_0514_with_building_ids\utils.pyc in lcm_estimate(cfg, choosers, chosen_fname, buildings, nodes)
131 chosen_fname,
132 alternatives,
--> 133 cfg)
134
135

C:\Anaconda\lib\site-packages\urbansim\models\lcm.pyc in fit_from_cfg(cls, choosers, chosen_fname, alternatives, cfgname)
1573 logger.debug('start: fit from configuration {}'.format(cfgname))
1574 lcm = cls.from_yaml(str_or_buffer=cfgname)
-> 1575 lcm.fit(choosers, alternatives, choosers[chosen_fname])
1576 for k, v in lcm._group.models.items():
1577 print("LCM RESULTS FOR SEGMENT %s\n" % str(k))

C:\Anaconda\lib\site-packages\urbansim\models\lcm.pyc in fit(self, choosers, alternatives, current_choice)
1302 self.add_segment(x)
1303
-> 1304 results = self._group.fit(choosers, alternatives, current_choice)
1305 logger.debug(
1306 'finish: fit models in segmented LCM {}'.format(self.name))

C:\Anaconda\lib\site-packages\urbansim\models\lcm.pyc in fit(self, choosers, alternatives, current_choice)
908 return {
909 name: self.models[name].fit(df, alternatives, current_choice)
--> 910 for name, df in self._iter_groups(choosers)}
911
912 @Property

C:\Anaconda\lib\site-packages\urbansim\models\lcm.pyc in ((name, df))
908 return {
909 name: self.models[name].fit(df, alternatives, current_choice)
--> 910 for name, df in self._iter_groups(choosers)}
911
912 @Property

C:\Anaconda\lib\site-packages\urbansim\models\lcm.pyc in fit(self, choosers, alternatives, current_choice)
347
348 self.log_likelihoods, self.fit_parameters = mnl.mnl_estimate(
--> 349 model_design.as_matrix(), chosen, self.sample_size)
350 self.fit_parameters.index = model_design.columns
351

C:\Anaconda\lib\site-packages\urbansim\urbanchoice\mnl.pyc in mnl_estimate(data, chosen, numalts, GPU, coeffrange, weights, lcgrad, beta)
248 beta = bfgs_result[0]
249 stderr = mnl_loglik(
--> 250 beta, data, chosen, numalts, weights, stderr=1, lcgrad=lcgrad)
251
252 l0beta = np.zeros(numvars)

C:\Anaconda\lib\site-packages\urbansim\urbanchoice\mnl.pyc in mnl_loglik(beta, data, chosen, numalts, weights, lcgrad, stderr)
91 gradmat = data.multiply_by_row(gradmat.reshape(1, gradmat.size()))
92 gradmat.reshape(numvars, numalts * numobs)
---> 93 return get_standard_error(get_hessian(gradmat.get_mat()))
94
95 chosen.reshape(numalts, numobs)

C:\Anaconda\lib\site-packages\urbansim\urbanchoice\mnl.pyc in get_hessian(derivative)
54
55 def get_hessian(derivative):
---> 56 return np.linalg.inv(np.dot(derivative, np.transpose(derivative)))
57
58

C:\Anaconda\lib\site-packages\numpy\linalg\linalg.pyc in inv(a)
518 signature = 'D->D' if isComplexType(t) else 'd->d'
519 extobj = get_linalg_error_extobj(_raise_linalgerror_singular)
--> 520 ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
521 return wrap(ainv.astype(result_t))
522

C:\Anaconda\lib\site-packages\numpy\linalg\linalg.pyc in _raise_linalgerror_singular(err, flag)
88
89 def _raise_linalgerror_singular(err, flag):
---> 90 raise LinAlgError("Singular matrix")
91
92 def _raise_linalgerror_nonposdef(err, flag):

LinAlgError: Singular matrix

[fscottfoti]

I don't know exactly, but this is almost always because you have one column that is exactly the same as another column so the estimation is not identified. The most common case is that both columns end up as all zeros, but they can be the same, or one can be the same as the other times some factor, so like col_a = col_b * 2.0 or something. I intended to add a check for this but haven't had the time to add it.

[lisalan520]

Very helpful! This gives me a direction. I'll double check with my data.
Thank you very much!

[quintinvh]

@lisalan520 have you managed to solve this issue? We have been sitting with exactly the same issue and if you could give us any guidance as to what caused this for you, it would save us a lot of time.

[Nisha-C]

I too faced the same issue..turns out, one of the columns had a duplicate. removing that solved it!