BUG: fix issue with get_dummies not using sparse correct

The underlying problem is that inside of get_dummies when you say
sparse=True it only encodes the columns that are part of the dummies
transformation to SparseDataFrames. This would be fine except if you
reindex on one of the dense frames it causes block errors at a lower
level.

This should fix that by casting all columns as sparse if sparse=True

Author: hexgnu

doc/source/whatsnew/v0.23.0.txt

@@ -353,6 +353,7 @@ Reshaping
 - Bug in :func:`Series.rank` where ``Series`` containing ``NaT`` modifies the ``Series`` inplace (:issue:`18521`)
 - Bug in :func:`cut` which fails when using readonly arrays (:issue:`18773`)
 - Bug in :func:`Dataframe.pivot_table` which fails when the ``aggfunc`` arg is of type string.  The behavior is now consistent with other methods like ``agg`` and ``apply`` (:issue:`18713`)
+- Bug in :func:`get_dummies` which doesn't cast all columns as sparse which causes reindex to fail. (:issue:`18914`)
 
 
 Numeric

pandas/core/reshape/reshape.py

@@ -841,6 +841,10 @@ def check_len(item, name):
             with_dummies = []
         else:
             with_dummies = [data.drop(columns_to_encode, axis=1)]
+            if sparse:
+                with_dummies = [SparseDataFrame(with_dummies[0],
+                                                index=with_dummies.index,
+                                                default_fill_value=0)]
 
         for (col, pre, sep) in zip(columns_to_encode, prefix, prefix_sep):
 

pandas/tests/reshape/test_reshape.py

@@ -454,6 +454,14 @@ def test_dataframe_dummies_preserve_categorical_dtype(self, dtype):
 
             tm.assert_frame_equal(result, expected)
 
+    def test_get_dummies_sparsify_all_columns(self):
+        # GH18914
+        df = DataFrame.from_items([('GDP', [1, 2]), ('Nation', ['AB', 'CD'])])
+        df = get_dummies(df, columns=['Nation'], sparse=True)
+        df2 = df.reindex(columns=['GDP'])
+
+        tm.assert_index_equal(df.index, df2.index)
+
 
 class TestCategoricalReshape(object):