Motorola SM56 kludge

Here’s how I did it:

I was able to make Motorola’s binary-only driver work on my Mandrake 8.1 installation. Motorola basically says that they’ve tested it only on RedHat 7.1; out of the box, the kernel module will not work on kernels > 2.4.6 (?). Insmod’ing it out of the box would give a segmentation fault with the message

'kernel BUG at slab.c:1097!'

WHAT COULD HAVE BEEN DONE
Online, I found a page that outlined steps on how to patch the slab memory allocator to prevent the bug from occuring. However, I couldn’t compile kernel modules (something which I have yet to figure out why), so that option wasn’t available to me. The next best thing was to kludge the module.

WHAT I DID
From reading the kernel sources, as well as reading about it online, the module crashes because of the different flag format used in kernels 2.4.6 and below. So, I wrote a small function to translate the module’s kmalloc() and __vmalloc() calls to use the proper flag format. Then, taking a cue from Mark Spieth’s Fixscript (check out this page as well as the LinModem HOWTO), I used objcopy to point the module’s kmalloc() and __vmalloc() calls to my functions. I then linked in my function, and used the resulting module. Command lines are as follows:

% objcopy --redefine-sym kmalloc=kmalloc_hack --redefine-sym \<br> __vmalloc=vmalloc_hack sm56.o sm56_h.o
% gcc -D__KERNEL__ -DMODULE -Wall -O -fomit-frame-pointer -o \<br>
kmhack.o -c kmalloc_hack.c
% ld -r -o sm56_hh.o sm56_h.o kmhack.o

I then cp’d the kludged driver in place of the installed driver.

SYSTEM
Mandrake 8.1 (vitamin)
Kernel 2.4.8-26mdk
32MB RAM
Motorola SM56 PCI II Modem

DISCLAIMER: Your mileage may vary. It works for me, but it might not work for you.

SOURCE CODE (kmalloc_hack.c)

/* kmalloc_hack.c
* Hack to make the SM56 binary-only module work
* on >2.4.8 kernels
*
* AUTHOR: Jan Michael Ibanez
* v 0.1
*
* NOTE :
* This is only a hack. Your mileage may vary.
*/

#include <linux/slab.h>
#include <linux/vmalloc.h>

void * kmalloc_hack(size_t size, int flags)
{
   /* perform a kmalloc() call on behalf
    * of the module, but with the right
    * flags
    */
   void * retval;
   int newflags = GFP_USER | GFP_DMA;
   /*
    * FIXME : function should translate flags,
    * instead of assuming GFP_USER | GFP_DMA
    */
   retval = kmalloc(size, newflags);

   return retval;
}

void * vmalloc_hack(unsigned long size, int gfp_mask, pgprot_t prot)
{
   /* perform a __vmalloc() call on behalf
    * of the module.
    */
   int new_gfp_mask = GFP_USER | GFP_DMA

   /* FIXME : function should translate 
    * gfp_mask instead of assuming
    * GFP_USER | GFP_DMA
    */
   void * retval = __vmalloc(size, new_gfp_mask, prot);
   return retval;
}

Originally from here