fit_transform() returns features in different order to sklearn

[ryanviolet]

I am attempting to use your excellent library alongside m2cgen and an sklearn model.

The test data (and ultimately the live feature data) needs to be poly transformed prior to feeding into the m2cgen generated method.

It seems that MachineLearningPurePy.Poly_Features_Pure_Py.fit_transform returns the transformed values in a different order to sklearn.PolynomialFeatures.fit_transform.

I haven't been able to see if there is a parameter that allows for returning of the data in the same order. Or, perhaps you could please let me know how I might programatically translate the order to that of sklearn.

Here are the two outputs that have the same values just in a different order.

Poly sample pp as np array:
[[1.60156810e+07 1.59576532e+07 0.00000000e+00 0.00000000e+00
0.00000000e+00 5.80278298e+05 1.58998356e+07 0.00000000e+00
0.00000000e+00 0.00000000e+00 5.78175840e+05 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
2.10245760e+04 1.58422275e+07 0.00000000e+00 0.00000000e+00
0.00000000e+00 5.76081000e+05 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 2.09484000e+04
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 7.61760000e+02
1.57848281e+07 0.00000000e+00 0.00000000e+00 0.00000000e+00
5.73993750e+05 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 2.08725000e+04 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 7.59000000e+02 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 2.76000000e+01 1.57276367e+07 0.00000000e+00
0.00000000e+00 0.00000000e+00 5.71914062e+05 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
2.07968750e+04 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
7.56250000e+02 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 2.75000000e+01
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]]

Poly sample sk:
[[1.00000000e+00 2.76000000e+01 2.75000000e+01 0.00000000e+00
0.00000000e+00 0.00000000e+00 7.61760000e+02 7.59000000e+02
0.00000000e+00 0.00000000e+00 0.00000000e+00 7.56250000e+02
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 2.10245760e+04 2.09484000e+04 0.00000000e+00
0.00000000e+00 0.00000000e+00 2.08725000e+04 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
2.07968750e+04 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
5.80278298e+05 5.78175840e+05 0.00000000e+00 0.00000000e+00
0.00000000e+00 5.76081000e+05 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 5.73993750e+05
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 5.71914062e+05
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 1.60156810e+07 1.59576532e+07
0.00000000e+00 0.00000000e+00 0.00000000e+00 1.58998356e+07
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 1.58422275e+07 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 1.57848281e+07 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
1.57276367e+07 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]]

TIA

[ryanviolet]

Hi @ThomIves are there any thoughts here please? Many thanks

[ThomIves]

Hi Ryan, After much thought, I think this just doesn't matter. I had wanted them to be the same, but I do like the order that my routine returns. Thoughts? Thom Ives, Ph.D. https://thomives.github.io/ThomIves/, https://www.linkedin.com/in/thomives/ https://integratedmlai.com/, https://github.com/ThomIves Text 208.391.3585

…

On Fri, Dec 9, 2022 at 4:31 AM ryanviolet ***@***.***> wrote: Hi @ThomIves <https://github.com/ThomIves> are there any thoughts here please? Many thanks — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4Z6CP6LGPH5AOCVRUG3SLWMMKBDANCNFSM6AAAAAASO3BCFM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ryanviolet]

Hi Thom, Thanks for getting back to me and HNY!

The issue is that we are using m2cgen to convert our sklearn model to a pure python function (on the fly as required in the cloud where we have processing power and flexibility) and then deploy it onto our embedded devices in the field (where adding compiled packages is restricted). Then on each field device, we need to poly transform the input data to feed into the m2cgen function in the same order for the prediction to be correct.

Kind regards,
Ryan

[ThomIves]

Wow Ryan, I didn't even know anyone was using this class for real world apps. That's great to know. Do you need the same order as SKLearn for some reason? As far as I know, my class will give you the same order each time. Perhaps I am misunderstanding what is needed on your end? Thom Ives, Ph.D. https://thomives.github.io/ThomIves/, https://www.linkedin.com/in/thomives/ https://integratedmlai.com/, https://github.com/ThomIves Text 208.391.3585

…

On Tue, Jan 3, 2023 at 4:34 AM ryanviolet ***@***.***> wrote: Hi Thom, Thanks for getting back to me and HNY! The issue is that we are using m2cgen <https://github.com/BayesWitnesses/m2cgen> to convert our sklearn model to a pure python function (on the fly as required in the cloud where we have processing power and flexibility) and then deploy it onto our embedded devices in the field (where adding compiled packages is restricted). Then on each field device, we need to poly transform the input data to feed into the m2cgen function in the same order for the prediction to be correct. Kind regards, Ryan — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4Z6CIUUQ7BQEPEAAFR65LWQQFEPANCNFSM6AAAAAASO3BCFM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ryanviolet]

Hi Thom - sorry for the delay. We aren't using it just yet but I've been putting together a POC for our software engineers to implement. Yes, we need the same order as SKLearn. The reason is, we are using SKLearn as our main model. But the poly transformation is being done on a device that doesn't support SKLearn. So we need to do the transform with your library, then feed it into the SKLearn model, does that make sense?

…

On Tue, Jan 3, 2023 at 5:57 PM Thom Ives ***@***.***> wrote: Wow Ryan, I didn't even know anyone was using this class for real world apps. That's great to know. Do you need the same order as SKLearn for some reason? As far as I know, my class will give you the same order each time. Perhaps I am misunderstanding what is needed on your end? Thom Ives, Ph.D. https://thomives.github.io/ThomIves/, https://www.linkedin.com/in/thomives/ https://integratedmlai.com/, https://github.com/ThomIves Text 208.391.3585 On Tue, Jan 3, 2023 at 4:34 AM ryanviolet ***@***.***> wrote: > Hi Thom, Thanks for getting back to me and HNY! > > The issue is that we are using m2cgen > <https://github.com/BayesWitnesses/m2cgen> to convert our sklearn model > to a pure python function (on the fly as required in the cloud where we > have processing power and flexibility) and then deploy it onto our embedded > devices in the field (where adding compiled packages is restricted). Then > on each field device, we need to poly transform the input data to feed into > the m2cgen function in the same order for the prediction to be correct. > > Kind regards, > Ryan > > — > Reply to this email directly, view it on GitHub > < #1 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AG4Z6CIUUQ7BQEPEAAFR65LWQQFEPANCNFSM6AAAAAASO3BCFM > > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <https://t.sidekickopen10.com/s3t/c/5/f18dQhb0S7kF8c7M27W1dbbvy59hl3kW7_k2841CXdp3MP1rKnt_y5GW2bzNMz5y_Gwt101?te=W3R5hFj4cm2zwW3P28X24hCPvhW43T4Nz3R3bFfW4kCJsw2RHCxCW3T1jZY3K1LYgW3T1k7r43PsGbW3JWH6T3JWHsBW3T3qBc3K96zSW1hV0BX4hF6zGW3_X4wH4fdb43W20W5J-1Z0lH70&si=8000000005670850&pi=969bfa65-2bd7-40cd-fcdc-82b3076d24ff>, or unsubscribe <https://t.sidekickopen10.com/s3t/c/5/f18dQhb0S7kF8c7M27W1dbbvy59hl3kW7_k2841CXdp3MP1rKnt_y5GW2bzNMz5y_Gwt101?te=W3R5hFj4cm2zwW3P28X24hCPvhW43T4P243X5bgW3SYLpP3T1MdZW1LFyV_4hCWypW49M7TJ1Gy80dW3Qzvpz2171XvW2MDbmb3dcqHWW1ZcDf91Sgl8vW1Vjc5M32gY5wW2TqDYK32mv53W2nMHjk2nWb3Nf2qM3bJ04&si=8000000005670850&pi=969bfa65-2bd7-40cd-fcdc-82b3076d24ff> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Rithum is a trading name of Violet Ultra Ltd, a company registered in England & Wales with company number 12809908 Registered Address: 128 City Road, London, EC1V 2NX, England

[ThomIves]

Hi Ryan, It does now. Two thoughts, I can try to replicate the order, or you guys could use it for dev and production ... ? Thom Ives, Ph.D. https://thomives.github.io/ThomIves/, https://www.linkedin.com/in/thomives/ https://integratedmlai.com/, https://github.com/ThomIves Text 208.391.3585

…

On Thu, Jan 5, 2023 at 9:13 AM ryanviolet ***@***.***> wrote: Hi Thom - sorry for the delay. We aren't using it just yet but I've been putting together a POC for our software engineers to implement. Yes, we need the same order as SKLearn. The reason is, we are using SKLearn as our main model. But the poly transformation is being done on a device that doesn't support SKLearn. So we need to do the transform with your library, then feed it into the SKLearn model, does that make sense? On Tue, Jan 3, 2023 at 5:57 PM Thom Ives ***@***.***> wrote: > Wow Ryan, I didn't even know anyone was using this class for real world > apps. That's great to know. Do you need the same order as SKLearn for some > reason? As far as I know, my class will give you the same order each time. > Perhaps I am misunderstanding what is needed on your end? > > Thom Ives, Ph.D. > https://thomives.github.io/ThomIves/, > https://www.linkedin.com/in/thomives/ > https://integratedmlai.com/, https://github.com/ThomIves > Text 208.391.3585 > > > On Tue, Jan 3, 2023 at 4:34 AM ryanviolet ***@***.***> wrote: > > > Hi Thom, Thanks for getting back to me and HNY! > > > > The issue is that we are using m2cgen > > <https://github.com/BayesWitnesses/m2cgen> to convert our sklearn model > > to a pure python function (on the fly as required in the cloud where we > > have processing power and flexibility) and then deploy it onto our > embedded > > devices in the field (where adding compiled packages is restricted). Then > > on each field device, we need to poly transform the input data to feed > into > > the m2cgen function in the same order for the prediction to be correct. > > > > Kind regards, > > Ryan > > > > — > > Reply to this email directly, view it on GitHub > > < > #1 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AG4Z6CIUUQ7BQEPEAAFR65LWQQFEPANCNFSM6AAAAAASO3BCFM > > > > . > > You are receiving this because you were mentioned.Message ID: > > ***@***.***> > > > > — > Reply to this email directly, view it on GitHub > < https://t.sidekickopen10.com/s3t/c/5/f18dQhb0S7kF8c7M27W1dbbvy59hl3kW7_k2841CXdp3MP1rKnt_y5GW2bzNMz5y_Gwt101?te=W3R5hFj4cm2zwW3P28X24hCPvhW43T4Nz3R3bFfW4kCJsw2RHCxCW3T1jZY3K1LYgW3T1k7r43PsGbW3JWH6T3JWHsBW3T3qBc3K96zSW1hV0BX4hF6zGW3_X4wH4fdb43W20W5J-1Z0lH70&si=8000000005670850&pi=969bfa65-2bd7-40cd-fcdc-82b3076d24ff >, > or unsubscribe > < https://t.sidekickopen10.com/s3t/c/5/f18dQhb0S7kF8c7M27W1dbbvy59hl3kW7_k2841CXdp3MP1rKnt_y5GW2bzNMz5y_Gwt101?te=W3R5hFj4cm2zwW3P28X24hCPvhW43T4P243X5bgW3SYLpP3T1MdZW1LFyV_4hCWypW49M7TJ1Gy80dW3Qzvpz2171XvW2MDbmb3dcqHWW1ZcDf91Sgl8vW1Vjc5M32gY5wW2TqDYK32mv53W2nMHjk2nWb3Nf2qM3bJ04&si=8000000005670850&pi=969bfa65-2bd7-40cd-fcdc-82b3076d24ff > > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- Rithum is a trading name of Violet Ultra Ltd, a company registered in England & Wales with company number 12809908 Registered Address: 128 City Road, London, EC1V 2NX, England — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4Z6CIUWZM56ZRSNTPL4M3WQ3XJJANCNFSM6AAAAAASO3BCFM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ryanviolet]

Thanks Thom - Sorry for the delay, I have passed this over to my CTO who might reach out if he needs to, I hope that's ok. Is there perhaps an easy way to contact you directly rather than a public forum?

Cheers,
Ryan

[ThomIves]

***@***.*** Thom Ives, Ph.D. https://thomives.github.io/ThomIves/, https://www.linkedin.com/in/thomives/ https://integratedmlai.com/, https://github.com/ThomIves Text 208.391.3585

…

On Thu, Jan 26, 2023 at 4:13 AM ryanviolet ***@***.***> wrote: Thanks Thom - Sorry for the delay, I have passed this over to my CTO who might reach out if he needs to, I hope that's ok. Is there perhaps an easy way to contact you directly rather than a public forum? Cheers, Ryan — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4Z6COG76JN2VJ7VIFRFIDWUJL5RANCNFSM6AAAAAASO3BCFM> . You are receiving this because you were mentioned.Message ID: ***@***.***>